This Rmarkdown file assesses the output of CheckV, DeepVirFinder, Kaiju, VIBRANT, VirSorter, and VirSorter2 on multiple training sets of microbial DNA, primarily from NCBI. Created from fungal, viral, bacterial, archeael, protist, and plasmid DNA sequences

Please reach out to James Riddell () or Bridget Hegarty () regarding any issues, or open an issue on github.

library(pROC)
Type 'citation("pROC")' for a citation.

Attaching package: ‘pROC’

The following objects are masked from ‘package:stats’:

    cov, smooth, var

Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 

Import the file that combines the results from each of the tools from running “combining_tool_output.Rmd”:

viruses <- read_tsv("../IntermediaryFiles/viral_tools_combined.tsv")

── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  seqtype = col_character(),
  contig = col_character(),
  checkv_provirus = col_character(),
  checkv_quality = col_character(),
  method.x = col_character(),
  Classified = col_character(),
  IDs_all = col_character(),
  Seq = col_character(),
  Kaiju_Viral = col_character(),
  Kingdom = col_character(),
  type = col_character(),
  vibrant_quality = col_character(),
  method.y = col_character(),
  vibrant_prophage = col_character(),
  vs2type = col_character(),
  max_score_group = col_character()
)
ℹ Use `spec()` for the full column specifications.

This section defines a viralness score “keep_score” based on the tool classifications. A final keep_score above 1 indicates we will keep that sequence and call it viral.

VIBRANT Quality == “High Quality Draft”: +1 Quality == “Medium Quality Draft”: +1 Quality == “Low Quality Draft” & provirus == TRUE: +0.5

Virsorter2 Viral >= 50: +0.5 Viral >= 0.95: +0.5

Virsorter category == 1,2,4,5: +1 category == 3,6: +0.5

DeepVirFinder: Score >= 0.7: +0.5 Score >= 0.9: +0.5

Kaiju: Kaiju_viral = “cellular organisms”: -1 Kaiju_viral = “Viruses”: +1

CheckV If %unknown >= 75: +0.5 Hallmark > 2: +1 viral_genes == 0 and host_genes >= 1: keep_score = 0 If 3*viral_genes <= host_genes: keep_score = 0 If length > 50,000 and hallmark == 0: keep_score = 0

This script produces visualizations of these combined viral scorings and includes ecological metrics like alpha diversity.

You can decide which combination is appropriate for them and only need use the tools appropriate for your data.

getting_viral_set_1 <- function(input_seqs,
There were 24 warnings (use warnings() to see them)
                                include_vibrant=FALSE, 
                                include_virsorter2=FALSE,
                                include_deepvirfinder=FALSE,
                                include_tuning=FALSE,
                                include_kaiju=FALSE,
                                include_virsorter=FALSE) {
  
  keep_score <- rep(0, nrow(input_seqs))
  
  if (include_vibrant) {
    keep_score[input_seqs$vibrant_quality=="high quality draft"] <- keep_score[input_seqs$vibrant_quality=="high quality draft"] + 1
    keep_score[input_seqs$vibrant_quality=="medium quality draft"] <- keep_score[input_seqs$vibrant_quality=="medium quality draft"] + 1
    keep_score[input_seqs$vibrant_quality=="low quality draft" & input_seqs$checkv_provirus=="Yes"] <- keep_score[input_seqs$vibrant_quality=="low quality draft" & input_seqs$checkv_provirus=="Yes"] + 0.5
#    keep_score[input_seqs$vibrant_quality=="low quality draft"] <- keep_score[input_seqs$vibrant_quality=="low quality draft"] + 0.5
  }
  
  if (include_virsorter2) {
    keep_score[input_seqs$viral>=50] <- keep_score[input_seqs$viral>=50] + 0.5
    keep_score[input_seqs$viral>=95] <- keep_score[input_seqs$viral>=95] + 0.5
  }
  
  if (include_virsorter) {
    keep_score[input_seqs$category==1] <- keep_score[input_seqs$category==1] + 1
    keep_score[input_seqs$category==2] <- keep_score[input_seqs$category==2] + 1
    keep_score[input_seqs$category==3] <- keep_score[input_seqs$category==3] + 0.5
    keep_score[input_seqs$category==4] <- keep_score[input_seqs$category==4] + 1
    keep_score[input_seqs$category==5] <- keep_score[input_seqs$category==5] + 1
    keep_score[input_seqs$category==6] <- keep_score[input_seqs$category==6] + 0.5
  }
  
  if (include_deepvirfinder) {
    keep_score[input_seqs$score>=0.7 & input_seqs$checkv_length<20000] <- keep_score[input_seqs$score>=0.7 & input_seqs$checkv_length<20000] + 0.5
   keep_score[input_seqs$score>=0.9 & input_seqs$checkv_length<20000] <- keep_score[input_seqs$score>=0.9 & input_seqs$checkv_length<20000] + 0.5
  }
  
  if (include_kaiju) {
    keep_score[input_seqs$Kaiju_Viral=="cellular organisms"] <- keep_score[input_seqs$Kaiju_Viral=="cellular organisms"] - 1
    keep_score[input_seqs$Kaiju_Viral=="Viruses"] <- keep_score[input_seqs$Kaiju_Viral=="Viruses"] + 0.5
  }
  
  if (include_tuning) {
    keep_score[input_seqs$hallmark>2] <- keep_score[input_seqs$hallmark>2] + 1
    keep_score[input_seqs$checkv_host_genes>50 & input_seqs$checkv_provirus=="No"] <- keep_score[input_seqs$checkv_host_genes>50 & input_seqs$checkv_provirus=="No"] - 1
    keep_score[input_seqs$percent_unknown>=75 & input_seqs$checkv_length<50000] <- keep_score[input_seqs$percent_unknown>=75 & input_seqs$checkv_length<50000] + 0.5
    keep_score[input_seqs$percent_viral>=50] <- keep_score[input_seqs$percent_viral>=50] + 0.5
    #keep_score[input_seqs$hallmark>=(input_seqs$checkv_viral_genes/5)] <- keep_score[input_seqs$hallmark>=(input_seqs$checkv_viral_genes/5)] + 1 #add some ratio
    keep_score[input_seqs$checkv_viral_genes==0 & input_seqs$checkv_host_genes>=1] <- keep_score[input_seqs$checkv_viral_genes==0 & input_seqs$checkv_host_genes>=1] - 1
    keep_score[((input_seqs$checkv_viral_genes*3) <= input_seqs$checkv_host_genes) & input_seqs$checkv_provirus=="No"] <- keep_score[((input_seqs$checkv_viral_genes*3) <= input_seqs$checkv_host_genes) & input_seqs$checkv_provirus=="No"] - 1 # consider accounting for provirus designation
#    keep_score[(input_seqs$checkv_viral_genes*3) <= input_seqs$checkv_host_genes] <- 0 # consider accounting for provirus designation
    keep_score[input_seqs$checkv_length>500000 & input_seqs$hallmark==0] <- keep_score[input_seqs$checkv_length>500000 & input_seqs$hallmark==0] - 1
  }
  
  return(keep_score)
  
}

Assessing performance against the “truth”

note that this is only as accurate as the annotations of the input sequences

this function calculates the precision, recall, and F1 score for each pipeline

assess_performance <- function(seqtype, keep_score) {
  
  truepositive <- rep("not viral", length(seqtype))
  truepositive[seqtype=="virus"] <- "viral"
  
  #make confusion matrix
  confusion_matrix <- rep("true negative", length(keep_score))
  confusion_matrix[truepositive=="viral" & keep_score<=1] <- "false negative"
  confusion_matrix[truepositive=="viral" & keep_score>=1] <- "true positive"
  confusion_matrix[truepositive=="not viral" & keep_score>=1] <- "false positive"
  
  TP <- table(confusion_matrix)[4]
  FP <- table(confusion_matrix)[2]
  TN <- table(confusion_matrix)[3]
  FN <- table(confusion_matrix)[1]
  
  precision <- TP/(TP+FP)
  recall <- TP/(TP+FN)
  F1 <- 2*precision*recall/(precision+recall)
  
  MCC <- (TP*TN-FP*FN)/sqrt(as.numeric(TP+FP)*as.numeric(TP+FN)*as.numeric(TN+FP)*as.numeric(TN+FN))
  
  auc <- round(auc(truepositive, keep_score),4)
  
  #by type metrics
  fungal_FP <- table(confusion_matrix[seqtype=="fungi"])[2]
  protist_FP <- table(confusion_matrix[seqtype=="protist"])[2]
  bacterial_FP <- table(confusion_matrix[seqtype=="bacteria"])[2]
  viral_FN <- table(confusion_matrix[seqtype=="virus"])[1]
  
  performance <- c(precision, recall, F1, MCC, auc, fungal_FP, 
                   protist_FP, bacterial_FP, viral_FN)
  names(performance) <- c("precision", "recall", "F1", "MCC", "AUC", "fungal_FP",
                          "protist_FP", "bacterial_FP", "viral_FN")
  
  return(performance)
}

combination of tools list

combos_list <- data.frame(toolcombo=rep(0, 64),
                          CheckV=rep(0, 64),
                          DVF=rep(0, 64),
                          Kaiju=rep(0, 64),
                          VIBRANT=rep(0, 64),
                          VS=rep(0, 64),
                          VS2=rep(0, 64))
p <- 1

for (i in c(0,1)){
  for (j in c(0,1)){
    for (k in c(0,1)){
      for (l in c(0,1)){
        for (m in c(0,1)){
          for (n in c(0,1)){
            combos_list$toolcombo[p] <- paste(i,j,k,l,m,n)
            combos_list$toolcombo2[p] <- paste(if(i){"C"}else{"0"},if(j){"DVF"}else{"0"},
                                               if(k){"K"}else{"0"},if(l){"VB"}else{"0"},
                                               if(m){"VS"}else{"0"},if(n){"VS2"}else{"0"})
            combos_list$CheckV[p] <- i
            combos_list$DVF[p] <- j
            combos_list$Kaiju[p] <- k
            combos_list$VIBRANT[p] <- l
            combos_list$VS[p] <- m
            combos_list$VS2[p] <- n
            p <- p+1
          }
        }
      }
    }
  }
}

combos_list <- combos_list[-1,]

this function builds a list of all of the combinations that the user wants to test. In this case, we’re comparing the performance of all unique combinations of the six tools.

build_score_list <- function(input_seqs, combos) {
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
  output <- data.frame(precision=rep(0, nrow(combos)),
                       recall=rep(0, nrow(combos)),
                       F1=rep(0, nrow(combos)),
                       MCC=rep(0, nrow(combos)),
                       AUC=rep(0, nrow(combos)),
                       fungal_FP=rep(0, nrow(combos)),
                       protist_FP=rep(0, nrow(combos)),
                       bacterial_FP=rep(0, nrow(combos)),
                       viral_FN=rep(0, nrow(combos)))
  for (i in 1:nrow(combos)) {
    keep_score <- getting_viral_set_1(input_seqs, include_vibrant = combos$VIBRANT[i],
                                            include_virsorter = combos$VS[i],
                                            include_virsorter2 = combos$VS2[i],
                                            include_tuning = combos$CheckV[i],
                                            include_kaiju = combos$Kaiju[i],
                                            include_deepvirfinder = combos$DVF[i])
  
    output[i,1:9] <- assess_performance(input_seqs$seqtype, keep_score)
    
    output$toolcombo[i] <- paste(combos$CheckV[i],combos$DVF[i],
                                 combos$Kaiju[i], combos$VIBRANT[i],
                                 combos$VS[i], combos$VS2[i])
  }
  
  output[is.na(output)] <- 0

  return (output)
}

Calculate the performance of each pipeline

accuracy_scores <- data.frame(testing_set_index=rep(0, nrow(combos_list)*10),
                      precision=rep(0, nrow(combos_list)*10),
                       recall=rep(0, nrow(combos_list)*10),
                       F1=rep(0, nrow(combos_list)*10),
                       MCC=rep(0, nrow(combos_list)*10), 
                      AUC=rep(0, nrow(combos_list)*10),
                      fungal_FP=rep(0, nrow(combos_list)*10),
                      protist_FP=rep(0, nrow(combos_list)*10),
                      bacterial_FP=rep(0, nrow(combos_list)*10),
                      viral_FN=rep(0, nrow(combos_list)*10))

accuracy_scores <- cbind(testing_set_index=rep(1, nrow(combos_list)),
                              build_score_list(viruses[viruses$Index==1,], combos_list))
for (i in 2:10) {
  accuracy_scores <- rbind(accuracy_scores,
                           cbind(testing_set_index=rep(i, nrow(combos_list)),
                              build_score_list(viruses[viruses$Index==i,], combos_list)))
}
library("stringr")
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
accuracy_scores$numtools <- str_count(accuracy_scores$toolcombo, "1")
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
#accuracy_scores <- accuracy_scores[order(accuracy_scores$numtools, decreasing=F),]
accuracy_scores <- accuracy_scores[order(accuracy_scores$MCC, decreasing=F),]
accuracy_scores$toolcombo <- factor(accuracy_scores$toolcombo, levels = unique(accuracy_scores$toolcombo))
accuracy_scores$numtools <- as.factor(accuracy_scores$numtools)

Visualize how the precision, recall, and F1 scores change across pipelines.

pal <- ggthemes::tableau_color_pal(palette="Tableau 10", type="regular")
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
p2 <- ggplot(accuracy_scores, aes(x=toolcombo, y=F1, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("F1 Score")
p2

ggplot(accuracy_scores, aes(x=toolcombo, y=precision, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Precision")

ggplot(accuracy_scores, aes(x=toolcombo, y=recall, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Recall")

ggplot(accuracy_scores, aes(x=precision, y=recall, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Precision") +
  ylab("Recall")

ggplot(accuracy_scores, aes(x=toolcombo, y=abs(precision-recall), 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Precision-Recall")

ggplot(accuracy_scores, aes(x=toolcombo, y=MCC, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("MCC")

ggplot(accuracy_scores, aes(x=toolcombo, y=AUC, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("AUC")

ggplot(accuracy_scores, aes(x=toolcombo, y=fungal_FP, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Fungal False Positives")


ggplot(accuracy_scores, aes(x=toolcombo, y=protist_FP, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Protist False Positives")


ggplot(accuracy_scores, aes(x=toolcombo, y=bacterial_FP, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Bacterial False Positives")

ggplot(accuracy_scores, aes(x=toolcombo, y=viral_FN, 
                                  color=numtools, fill=numtools)) +
  geom_point(alpha=0.5) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Viral False Negatives")

write_tsv(accuracy_scores, "20220927_accuracy_scores.tsv")

to do: add in clustering and ordination like in the drinking water R notebook

Experimenting

high precision example

viruses$keep_score_high_precision <- getting_viral_set_1(viruses, include_deepvirfinder = F,
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
                                              include_vibrant = T,
                                              include_virsorter2 = F,
                                              include_kaiju = T,
                                              include_tuning = T,
                                              include_virsorter = F)
viruses$confusion_matrix_high_precision <- "true negative"
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
viruses$confusion_matrix_high_precision[viruses$seqtype=="virus" & viruses$keep_score_high_precision<1] <- "false negative"
viruses$confusion_matrix_high_precision[viruses$seqtype=="virus" & viruses$keep_score_high_precision>=1] <- "true positive"
viruses$confusion_matrix_high_precision[viruses$seqtype!="virus" & viruses$keep_score_high_precision>=1] <- "false positive"

visualizing confusion matrix by taxa

confusion_by_taxa <- melt(table(viruses$confusion_matrix_high_precision, viruses$seqtype, viruses$Index))
The melt generic in data.table has been passed a table and will attempt to redirect to the relevant reshape2 method; please note that reshape2 is deprecated, and this redirection is now deprecated as well. To continue using melt methods from reshape2 while both libraries are attached, e.g. melt.list, you can prepend the namespace like reshape2::melt(table(viruses$confusion_matrix_high_precision, viruses$seqtype,     viruses$Index)). In the next version, this warning will become an error.
colnames(confusion_by_taxa) <- c("confusion_matrix", "seqtype","Index", "count")
pal <- ggthemes::tableau_color_pal(palette="Tableau 10", type="regular")
Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
2: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
3: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
4: Unknown or uninitialised column: `keep_score_visualize`. 
5: Unknown or uninitialised column: `truepositive`. 
ggplot(confusion_by_taxa, aes(x=count, y=as.factor(Index),
                   fill=confusion_matrix,
                   color=confusion_matrix)) +
  geom_bar(stat="identity") +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Sequences") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()

 ggplot(viruses, aes(x=checkv_viral_genes, y=confusion_matrix_high_precision,
                   fill=confusion_matrix_high_precision,
                   color=confusion_matrix_high_precision)) +
  geom_boxplot(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Viral Sequences") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()


 ggplot(viruses, aes(x=percent_viral, y=confusion_matrix_high_precision,
                   fill=confusion_matrix_high_precision,
                   color=confusion_matrix_high_precision)) +
  geom_boxplot(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Percent Genes Viral") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()


 ggplot(viruses, aes(x=hallmark, y=confusion_matrix_high_precision,
                   fill=confusion_matrix_high_precision,
                   color=confusion_matrix_high_precision)) +
  geom_boxplot(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Hallmark Genes") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()

 
ggplot(viruses, aes(x=hallmark, y=checkv_viral_genes,
                   fill=confusion_matrix_high_precision,
                   color=confusion_matrix_high_precision)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Hallmark Genes") +
  ylab("Number of Viral Genes") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()

viruses_false_positive <- viruses[viruses$confusion_matrix_high_precision=="false positive",]
viruses_false_negative <- viruses[viruses$confusion_matrix_high_precision=="false negative",]
ggplot(viruses, aes(x=hallmark, y=checkv_viral_genes,
                   fill=checkv_length,
                   color=checkv_length,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Number of Viral Genes") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()


ggplot(viruses_false_positive, aes(x=hallmark, y=checkv_length,
                   fill=checkv_viral_genes,
                   color=checkv_viral_genes,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()


ggplot(viruses_false_positive[viruses_false_positive$seqtype=="bacteria"], aes(x=hallmark, y=checkv_length,
                   fill=checkv_viral_genes,
                   color=checkv_viral_genes,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~Kaiju_Viral, scales = "free") +
  coord_flip()


ggplot(viruses_false_positive[viruses_false_positive$seqtype=="fungi"], aes(x=hallmark, y=checkv_length,
                   fill=keep_score_high_precision,
                   color=keep_score_high_precision,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~Kaiju_Viral, scales = "free") +
  coord_flip()


ggplot(viruses_false_positive[viruses_false_positive$seqtype=="protist"], aes(x=hallmark, y=checkv_length,
                   fill=checkv_viral_genes,
                   color=checkv_viral_genes,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~Kaiju_Viral, scales = "free") +
  coord_flip()


ggplot(viruses_false_negative, aes(x=hallmark, y=checkv_length,
                   fill=checkv_viral_genes,
                   color=checkv_viral_genes,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~Kaiju_Viral, scales = "free") +
  coord_flip()


ggplot(viruses_false_negative, aes(x=hallmark, y=checkv_length,
                   fill=keep_score_high_precision,
                   color=keep_score_high_precision,
                   shape=checkv_provirus)) +
  geom_point(alpha=0.3) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Hallmark Genes") +
  ylab("Contig Length") + 
  facet_wrap(~Kaiju_Viral, scales = "free") +
  coord_flip()

table(viruses$hallmark[viruses$confusion_matrix_high_precision=="false positive"]>0)

FALSE  TRUE 
 4398  2608 
table(viruses$percent_host[viruses$confusion_matrix_high_precision=="false positive"]<50)

FALSE  TRUE 
  855  6151 

high MCC example

viruses$keep_score_high_MCC <- getting_viral_set_1(viruses, include_deepvirfinder = F,
There were 41 warnings (use warnings() to see them)
                                              include_vibrant = T,
                                              include_virsorter2 = F,
                                              include_kaiju = T,
                                              include_tuning = T,
                                              include_virsorter = T)
viruses$confusion_matrix_high_MCC <- "true negative"
There were 22 warnings (use warnings() to see them)
viruses$confusion_matrix_high_MCC[viruses$seqtype=="virus" & viruses$keep_score_high_MCC<1] <- "false negative"
viruses$confusion_matrix_high_MCC[viruses$seqtype=="virus" & viruses$keep_score_high_MCC>=1] <- "true positive"
viruses$confusion_matrix_high_MCC[viruses$seqtype!="virus" & viruses$keep_score_high_MCC>=1] <- "false positive"

visualizing confusion matrix by taxa

confusion_by_taxa <- melt(table(viruses$confusion_matrix_high_MCC, viruses$seqtype, viruses$Index))
The melt generic in data.table has been passed a table and will attempt to redirect to the relevant reshape2 method; please note that reshape2 is deprecated, and this redirection is now deprecated as well. To continue using melt methods from reshape2 while both libraries are attached, e.g. melt.list, you can prepend the namespace like reshape2::melt(table(viruses$confusion_matrix_high_MCC, viruses$seqtype, viruses$Index)). In the next version, this warning will become an error.Warning messages:
1: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
2: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
3: Unknown or uninitialised column: `confusion_matrix_high_precision`. 
4: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
5: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
6: Unknown or uninitialised column: `confusion_matrix_high_recall`. 
7: Unknown or uninitialised column: `keep_score_visualize`. 
8: Unknown or uninitialised column: `truepositive`. 
colnames(confusion_by_taxa) <- c("confusion_matrix", "seqtype","Index", "count")
pal <- ggthemes::tableau_color_pal(palette="Tableau 10", type="regular")
ggplot(confusion_by_taxa, aes(x=count, y=as.factor(Index),
                   fill=confusion_matrix,
                   color=confusion_matrix)) +
  geom_bar(stat="identity") +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Sequences") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()

high recall example

viruses$keep_score_high_recall <- getting_viral_set_1(viruses, include_deepvirfinder = T,
There were 15 warnings (use warnings() to see them)
                                              include_vibrant = T,
                                              include_virsorter2 = T,
                                              include_kaiju = T,
                                              include_tuning = T,
                                              include_virsorter = T)
viruses$confusion_matrix_high_recall <- "true negative"
viruses$confusion_matrix_high_recall[viruses$seqtype=="virus" & viruses$keep_score_high_recall<1] <- "false negative"
viruses$confusion_matrix_high_recall[viruses$seqtype=="virus" & viruses$keep_score_high_recall>=1] <- "true positive"
viruses$confusion_matrix_high_recall[viruses$seqtype!="virus" & viruses$keep_score_high_recall>=1] <- "false positive"

visualizing confusion matrix by taxa

confusion_by_taxa <- melt(table(viruses$confusion_matrix_high_recall, viruses$seqtype, viruses$Index))
The melt generic in data.table has been passed a table and will attempt to redirect to the relevant reshape2 method; please note that reshape2 is deprecated, and this redirection is now deprecated as well. To continue using melt methods from reshape2 while both libraries are attached, e.g. melt.list, you can prepend the namespace like reshape2::melt(table(viruses$confusion_matrix_high_recall, viruses$seqtype,     viruses$Index)). In the next version, this warning will become an error.
colnames(confusion_by_taxa) <- c("confusion_matrix", "seqtype","Index", "count")
pal <- ggthemes::tableau_color_pal(palette="Tableau 10", type="regular")
Warning messages:
1: Unknown or uninitialised column: `keep_score_visualize`. 
2: Unknown or uninitialised column: `truepositive`. 
p2 <- ggplot(confusion_by_taxa, aes(x=count, y=as.factor(Index),
                   fill=confusion_matrix,
                   color=confusion_matrix)) +
  geom_bar(stat="identity") +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(4)), 0.5),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(4)), 1),
                    labels=c("false negative", "false positive", 
                             "true negative", "true positive")) +
  xlab("Number of Sequences") +
  ylab("") + 
  facet_wrap(~seqtype, scales = "free") +
  coord_flip()
p2

looking at false negatives

viruses_false_negs <- viruses[(viruses$seqtype=="virus" & viruses$keep_score_high_recall<=1),]

Visualizing confusion matrix by number of tools

viruses$keep_score_visualize <- viruses$keep_score
viruses$keep_score_visualize[viruses$keep_score>1] <- ">1"
viruses$keep_score_visualize <- factor(viruses$keep_score_visualize, 
                                       levels=c("-0.5", "-1", "0", "0.5","1", ">1"))
viruses$keep_score_visualize <- factor(viruses$keep_score_visualize, 
                                       labels=c("≤ 0", "≤ 0", "≤ 0", "0.5","1", "> 1"))
levels(factor(viruses$keep_score_visualize))
pal <- ggthemes::tableau_color_pal(palette="Tableau 20", type="regular")
p1 <- ggplot(viruses, aes(x=as.factor(Index),
                   fill=keep_score_visualize, color=keep_score_visualize)) +
  geom_bar(stat="count", position="stack") +
  theme_light() +
  coord_flip() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16)
  ) +
  scale_color_manual(name = 'Viral Score',
                     values = alpha(c(pal(4)), 1)) +
  scale_fill_manual(name = 'Viral Score',
                     values = alpha(c(pal(4)), 0.5)) +
  xlab("Index") +
  ylab("Sequence Count") +
  facet_wrap(~confusion_matrix, scales = "free")
p1

ROC

library(pROC)
viruses$truepositive <- rep(0, nrow(viruses))
viruses$truepositive[viruses$seqtype=="virus"] <- 1
rocobj <- roc(viruses$truepositive, viruses$keep_score)
rocobj_all <- roc(viruses$truepositive, viruses$keep_score_all)
auc <- round(auc(viruses$truepositive, viruses$keep_score),4)
auc_all <- round(auc(viruses$truepositive, viruses$keep_score_all),4)
#create ROC plot
ggroc(rocobj, colour = 'steelblue', size = 2) +
  ggtitle(paste0('ROC Curve ', '(AUC = ', auc, ')')) +
  coord_equal()
ggroc(rocobj_all, colour = 'green', size = 2) +
  ggtitle(paste0('ROC Curve ', '(AUC = ', auc_all, ')'))

Sensitivity: The probability that the model predicts a positive outcome for an observation when indeed the outcome is positive. Specificity: The probability that the model predicts a negative outcome for an observation when indeed the outcome is negative.

Comparing behavior of all testing sets combined (clustering analyses)

viral_scores <- matrix(data=0, nrow=nrow(viruses), ncol=nrow(combos_list))
num_viruses <- data.frame(toolcombo=rep(0, nrow(combos_list)),
                          num_viruses=rep(0, nrow(combos_list)))

for (i in 1:nrow(combos_list)) {
  viral_scores[,i] <- getting_viral_set_1(viruses, include_vibrant = combos_list$VIBRANT[i],
                                            include_virsorter = combos_list$VS[i],
                                            include_virsorter2 = combos_list$VS2[i],
                                            include_tuning = combos_list$CheckV[i],
                                            include_kaiju = combos_list$Kaiju[i],
                                            include_deepvirfinder = combos_list$DVF[i])
  
  num_viruses$num_viruses[i] <- table(viral_scores[,i]>=1)[[2]]
  
  num_viruses$toolcombo[i] <- combos_list$toolcombo[i]
  
  num_viruses$toolcombo2[i] <- combos_list$toolcombo2[i]
}

num_viruses$numtools <- str_count(num_viruses$toolcombo, "1")
num_viruses <- num_viruses[order(num_viruses$num_viruses, decreasing=F),]
num_viruses$toolcombo <- factor(num_viruses$toolcombo, levels = unique(num_viruses$toolcombo))
num_viruses$toolcombo2 <- factor(num_viruses$toolcombo2, levels = unique(num_viruses$toolcombo2))
num_viruses$numtools <- as.factor(num_viruses$numtools)
ggplot(num_viruses, aes(x=toolcombo, y=num_viruses, 
                                  color=numtools, fill=numtools)) +
  geom_point() +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Num Viruses Predicted")

ggplot(num_viruses, aes(x=toolcombo2, y=num_viruses, 
                                  color=numtools, fill=numtools)) +
  geom_point() +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Tool Combination (CV, DVF, KJ, VB, VS, VS2)") +
  ylab("Num Viruses Predicted")
ggplot(num_viruses, aes(x=numtools, y=num_viruses)) +
  geom_boxplot(aes(color=numtools)) +
  geom_point(aes(color=numtools, fill=numtools)) +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14, angle = 90),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  xlab("Number of Tools") +
  ylab("Num Viruses Predicted")
viral_scores_nozeros <- viral_scores[rowSums(viral_scores)>0,]
viral_scores_nozeros <- viral_scores_nozeros + 1
viral_scores_nozeros <- as.data.frame(viral_scores_nozeros)

colnames(viral_scores_nozeros) <- num_viruses$toolcombo2
tooldata <- num_viruses

rownames(tooldata) <- tooldata$toolcombo2
tooldata <- num_viruses

rownames(tooldata) <- tooldata$toolcombo2
physeq_pooled <- phyloseq(otu_table(viral_scores_nozeros, taxa_are_rows = T),
                                     sample_data(tooldata))
ordination <- phyloseq::ordinate(physeq =physeq_pooled, method = "PCoA", distance = "bray")
phyloseq::plot_ordination(physeq = physeq_pooled, ordination = ordination,
                          shape="numtools", color="num_viruses") + 
  geom_point(size = 3) +
  theme_bw() +
  geom_label(label=tooldata$toolcombo)

phyloseq::plot_ordination(physeq = physeq_pooled, ordination = ordination,
                          shape="numtools", color="num_viruses") + 
  geom_point(size = 3) +
  theme_bw()

to do: try coloring above based on the F1 scores of the testing set on each combination

bray_dist <- phyloseq::distance(physeq_pooled, method="bray")
clusters <- hclust(dist(bray_dist))
plot(clusters)

myclusters <- cutree(clusters, h=1.1)
names(myclusters[myclusters==1])
names(myclusters[myclusters==2])
names(myclusters[myclusters==3])
names(myclusters[myclusters==4])
names(myclusters[myclusters==5])

myclusters_df <- tibble(combo=names(myclusters),
                            cluster_index=myclusters)

myclusters_df <- separate(myclusters_df, col=combo, into=c("CheckV", "DVF",
                                                            "Kaiju", "VIBRANT",
                                                            "VirSorter", "VirSorter2"),
                          sep=" ", remove = F)


tool_count <- as.data.frame(rbind(table(myclusters_df$CheckV, myclusters_df$cluster_index)[2,],
                          table(myclusters_df$DVF, myclusters_df$cluster_index)[2,],
                          table(myclusters_df$Kaiju, myclusters_df$cluster_index)[2,],
                          table(myclusters_df$VIBRANT, myclusters_df$cluster_index)[2,],
                          table(myclusters_df$VirSorter, myclusters_df$cluster_index)[2,],
                          table(myclusters_df$VirSorter2, myclusters_df$cluster_index)[2,])
                    )

tool_count$method <- c("CheckV", "DVF", "Kaiju", "VIBRANT", "VirSorter", "VirSorter2")

tool_count <- melt(tool_count)

colnames(tool_count) <- c("tool", "cluster_index", "tool_count")
pal <- ggthemes::tableau_color_pal(palette="Tableau 10", type="regular")

ggplot(tool_count, aes(x=cluster_index, y=tool_count,
                   fill=cluster_index,
                   color=cluster_index)) +
  geom_bar(stat="identity") +
  theme_light() +
  theme(
    panel.grid.major.y = element_blank(),
    panel.border = element_blank(),
    axis.ticks.y = element_blank(),
    legend.position = "bottom",
    axis.text.y=element_text(size=14),
    axis.text.x=element_text(size=14),
    legend.text=element_text(size=12),
    axis.title=element_text(size=16),
  ) +
  scale_fill_manual(name="",
                     values = alpha(rev(pal(6)), 0.5)) +
  scale_color_manual(name="",
                     values = alpha(rev(pal(6)), 1)) +
  xlab("Cluster") +
  ylab("Number of Times in Cluster") + 
  facet_wrap(~tool, scales = "free")
LS0tCnRpdGxlOiAiVmlyYWwgU2VxdWVuY2UgU29ydGluZyBUb29scyBFdmFsdWF0aW9uIgphdXRob3I6IEJyaWRnZXQgSGVnYXJ0eSwgSmFtZXMgUmlkZGVsbApkYXRlOiAwNy0yMi0yMDIyCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KVGhpcyBSbWFya2Rvd24gZmlsZSBhc3Nlc3NlcyB0aGUgb3V0cHV0IG9mIENoZWNrViwgRGVlcFZpckZpbmRlciwgS2FpanUsClZJQlJBTlQsIFZpclNvcnRlciwgYW5kIFZpclNvcnRlcjIgb24gbXVsdGlwbGUgdHJhaW5pbmcgc2V0cyBvZiBtaWNyb2JpYWwgRE5BLCAKcHJpbWFyaWx5IGZyb20gTkNCSS4gQ3JlYXRlZCBmcm9tIGZ1bmdhbCwgdmlyYWwsIGJhY3RlcmlhbCwgYXJjaGVhZWwsIHByb3Rpc3QsCmFuZCBwbGFzbWlkIEROQSBzZXF1ZW5jZXMKClBsZWFzZSByZWFjaCBvdXQgdG8gSmFtZXMgUmlkZGVsbCAocmlkZGVsbC4yNkBidWNrZXllbWFpbC5vc3UuZWR1KSBvcgpCcmlkZ2V0IEhlZ2FydHkgKGJlaDUzQGNhc2UuZWR1KSByZWdhcmRpbmcgYW55IGlzc3Vlcywgb3Igb3BlbiBhbiBpc3N1ZSBvbiBnaXRodWIuCgpgYGB7ciBzZXR1cC1saWJyYXJ5fQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkocGx5cikKbGlicmFyeShyZXNoYXBlMikKbGlicmFyeSh2aXJpZGlzKQpsaWJyYXJ5KHRpZHlyKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KHJlYWRyKQpsaWJyYXJ5KGRhdGEudGFibGUpCmxpYnJhcnkocFJPQykKYGBgCgpJbXBvcnQgdGhlIGZpbGUgdGhhdCBjb21iaW5lcyB0aGUgcmVzdWx0cyBmcm9tIGVhY2ggb2YgdGhlIHRvb2xzIGZyb20gcnVubmluZyAiY29tYmluaW5nX3Rvb2xfb3V0cHV0LlJtZCI6CmBgYHtyfQp2aXJ1c2VzIDwtIHJlYWRfdHN2KCIuLi9JbnRlcm1lZGlhcnlGaWxlcy92aXJhbF90b29sc19jb21iaW5lZC50c3YiKQpgYGAKClRoaXMgc2VjdGlvbiBkZWZpbmVzIGEgdmlyYWxuZXNzIHNjb3JlICJrZWVwX3Njb3JlIiBiYXNlZCBvbiB0aGUgdG9vbCBjbGFzc2lmaWNhdGlvbnMuIApBIGZpbmFsIGtlZXBfc2NvcmUgYWJvdmUgMSBpbmRpY2F0ZXMgd2Ugd2lsbCBrZWVwIHRoYXQgc2VxdWVuY2UgYW5kIGNhbGwgaXQgdmlyYWwuCgpWSUJSQU5UCiAgICBRdWFsaXR5ID09ICJIaWdoIFF1YWxpdHkgRHJhZnQiOiArMQogICAgUXVhbGl0eSA9PSAiTWVkaXVtIFF1YWxpdHkgRHJhZnQiOiArMQogICAgUXVhbGl0eSA9PSAiTG93IFF1YWxpdHkgRHJhZnQiICYgcHJvdmlydXMgPT0gVFJVRTogKzAuNQoKVmlyc29ydGVyMgogICAgVmlyYWwgPj0gNTA6ICswLjUKICAgIFZpcmFsID49IDAuOTU6ICswLjUKClZpcnNvcnRlcgogICAgY2F0ZWdvcnkgPT0gIDEsMiw0LDU6ICsxCiAgICBjYXRlZ29yeSA9PSAzLDY6ICswLjUKCkRlZXBWaXJGaW5kZXI6CiAgICBTY29yZSA+PSAwLjc6ICswLjUKICAgIFNjb3JlID49IDAuOTogKzAuNQoKS2FpanU6CiAgICBLYWlqdV92aXJhbCA9ICJjZWxsdWxhciBvcmdhbmlzbXMiOiAtMQogICAgS2FpanVfdmlyYWwgPSAiVmlydXNlcyI6ICsxCgpDaGVja1YKICAgIElmICV1bmtub3duID49IDc1OiArMC41CiAgICBIYWxsbWFyayA+IDI6ICsxCiAgICB2aXJhbF9nZW5lcyA9PSAwIGFuZCBob3N0X2dlbmVzID49IDE6IGtlZXBfc2NvcmUgPSAwCiAgICBJZiAzKnZpcmFsX2dlbmVzIDw9IGhvc3RfZ2VuZXM6IGtlZXBfc2NvcmUgPSAwCiAgICBJZiBsZW5ndGggPiA1MCwwMDAgYW5kIGhhbGxtYXJrID09IDA6IGtlZXBfc2NvcmUgPSAwCiAgICAKClRoaXMgc2NyaXB0IHByb2R1Y2VzIHZpc3VhbGl6YXRpb25zIG9mIHRoZXNlIGNvbWJpbmVkIHZpcmFsIHNjb3JpbmdzIGFuZAppbmNsdWRlcyBlY29sb2dpY2FsIG1ldHJpY3MgbGlrZSBhbHBoYSBkaXZlcnNpdHkuCgpZb3UgY2FuIGRlY2lkZSB3aGljaCBjb21iaW5hdGlvbiBpcyBhcHByb3ByaWF0ZSBmb3IgdGhlbSBhbmQgb25seSBuZWVkIHVzZSB0aGUKdG9vbHMgYXBwcm9wcmlhdGUgZm9yIHlvdXIgZGF0YS4KCmBgYHtyIGdldHRpbmdfdmlyYWxfc2V0XzF9CmdldHRpbmdfdmlyYWxfc2V0XzEgPC0gZnVuY3Rpb24oaW5wdXRfc2VxcywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpYnJhbnQ9RkFMU0UsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyMj1GQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX2RlZXB2aXJmaW5kZXI9RkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV90dW5pbmc9RkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV9rYWlqdT1GQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlcj1GQUxTRSkgewogIAogIGtlZXBfc2NvcmUgPC0gcmVwKDAsIG5yb3coaW5wdXRfc2VxcykpCiAgCiAgaWYgKGluY2x1ZGVfdmlicmFudCkgewogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJHZpYnJhbnRfcXVhbGl0eT09ImhpZ2ggcXVhbGl0eSBkcmFmdCJdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyR2aWJyYW50X3F1YWxpdHk9PSJoaWdoIHF1YWxpdHkgZHJhZnQiXSArIDEKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyR2aWJyYW50X3F1YWxpdHk9PSJtZWRpdW0gcXVhbGl0eSBkcmFmdCJdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyR2aWJyYW50X3F1YWxpdHk9PSJtZWRpdW0gcXVhbGl0eSBkcmFmdCJdICsgMQogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJHZpYnJhbnRfcXVhbGl0eT09ImxvdyBxdWFsaXR5IGRyYWZ0IiAmIGlucHV0X3NlcXMkY2hlY2t2X3Byb3ZpcnVzPT0iWWVzIl0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJHZpYnJhbnRfcXVhbGl0eT09ImxvdyBxdWFsaXR5IGRyYWZ0IiAmIGlucHV0X3NlcXMkY2hlY2t2X3Byb3ZpcnVzPT0iWWVzIl0gKyAwLjUKIyAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkdmlicmFudF9xdWFsaXR5PT0ibG93IHF1YWxpdHkgZHJhZnQiXSA8LSBrZWVwX3Njb3JlW2lucHV0X3NlcXMkdmlicmFudF9xdWFsaXR5PT0ibG93IHF1YWxpdHkgZHJhZnQiXSArIDAuNQogIH0KICAKICBpZiAoaW5jbHVkZV92aXJzb3J0ZXIyKSB7CiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkdmlyYWw+PTUwXSA8LSBrZWVwX3Njb3JlW2lucHV0X3NlcXMkdmlyYWw+PTUwXSArIDAuNQogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJHZpcmFsPj05NV0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJHZpcmFsPj05NV0gKyAwLjUKICB9CiAgCiAgaWYgKGluY2x1ZGVfdmlyc29ydGVyKSB7CiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkY2F0ZWdvcnk9PTFdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09MV0gKyAxCiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkY2F0ZWdvcnk9PTJdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09Ml0gKyAxCiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkY2F0ZWdvcnk9PTNdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09M10gKyAwLjUKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09NF0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNhdGVnb3J5PT00XSArIDEKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09NV0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNhdGVnb3J5PT01XSArIDEKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRjYXRlZ29yeT09Nl0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNhdGVnb3J5PT02XSArIDAuNQogIH0KICAKICBpZiAoaW5jbHVkZV9kZWVwdmlyZmluZGVyKSB7CiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkc2NvcmU+PTAuNyAmIGlucHV0X3NlcXMkY2hlY2t2X2xlbmd0aDwyMDAwMF0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJHNjb3JlPj0wLjcgJiBpbnB1dF9zZXFzJGNoZWNrdl9sZW5ndGg8MjAwMDBdICsgMC41CiAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRzY29yZT49MC45ICYgaW5wdXRfc2VxcyRjaGVja3ZfbGVuZ3RoPDIwMDAwXSA8LSBrZWVwX3Njb3JlW2lucHV0X3NlcXMkc2NvcmU+PTAuOSAmIGlucHV0X3NlcXMkY2hlY2t2X2xlbmd0aDwyMDAwMF0gKyAwLjUKICB9CiAgCiAgaWYgKGluY2x1ZGVfa2FpanUpIHsKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRLYWlqdV9WaXJhbD09ImNlbGx1bGFyIG9yZ2FuaXNtcyJdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRLYWlqdV9WaXJhbD09ImNlbGx1bGFyIG9yZ2FuaXNtcyJdIC0gMQogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJEthaWp1X1ZpcmFsPT0iVmlydXNlcyJdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRLYWlqdV9WaXJhbD09IlZpcnVzZXMiXSArIDAuNQogIH0KICAKICBpZiAoaW5jbHVkZV90dW5pbmcpIHsKICAgIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRoYWxsbWFyaz4yXSA8LSBrZWVwX3Njb3JlW2lucHV0X3NlcXMkaGFsbG1hcms+Ml0gKyAxCiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkY2hlY2t2X2hvc3RfZ2VuZXM+NTAgJiBpbnB1dF9zZXFzJGNoZWNrdl9wcm92aXJ1cz09Ik5vIl0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNoZWNrdl9ob3N0X2dlbmVzPjUwICYgaW5wdXRfc2VxcyRjaGVja3ZfcHJvdmlydXM9PSJObyJdIC0gMQogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJHBlcmNlbnRfdW5rbm93bj49NzUgJiBpbnB1dF9zZXFzJGNoZWNrdl9sZW5ndGg8NTAwMDBdIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRwZXJjZW50X3Vua25vd24+PTc1ICYgaW5wdXRfc2VxcyRjaGVja3ZfbGVuZ3RoPDUwMDAwXSArIDAuNQogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJHBlcmNlbnRfdmlyYWw+PTUwXSA8LSBrZWVwX3Njb3JlW2lucHV0X3NlcXMkcGVyY2VudF92aXJhbD49NTBdICsgMC41CiAgICAja2VlcF9zY29yZVtpbnB1dF9zZXFzJGhhbGxtYXJrPj0oaW5wdXRfc2VxcyRjaGVja3ZfdmlyYWxfZ2VuZXMvNSldIDwtIGtlZXBfc2NvcmVbaW5wdXRfc2VxcyRoYWxsbWFyaz49KGlucHV0X3NlcXMkY2hlY2t2X3ZpcmFsX2dlbmVzLzUpXSArIDEgI2FkZCBzb21lIHJhdGlvCiAgICBrZWVwX3Njb3JlW2lucHV0X3NlcXMkY2hlY2t2X3ZpcmFsX2dlbmVzPT0wICYgaW5wdXRfc2VxcyRjaGVja3ZfaG9zdF9nZW5lcz49MV0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNoZWNrdl92aXJhbF9nZW5lcz09MCAmIGlucHV0X3NlcXMkY2hlY2t2X2hvc3RfZ2VuZXM+PTFdIC0gMQogICAga2VlcF9zY29yZVsoKGlucHV0X3NlcXMkY2hlY2t2X3ZpcmFsX2dlbmVzKjMpIDw9IGlucHV0X3NlcXMkY2hlY2t2X2hvc3RfZ2VuZXMpICYgaW5wdXRfc2VxcyRjaGVja3ZfcHJvdmlydXM9PSJObyJdIDwtIGtlZXBfc2NvcmVbKChpbnB1dF9zZXFzJGNoZWNrdl92aXJhbF9nZW5lcyozKSA8PSBpbnB1dF9zZXFzJGNoZWNrdl9ob3N0X2dlbmVzKSAmIGlucHV0X3NlcXMkY2hlY2t2X3Byb3ZpcnVzPT0iTm8iXSAtIDEgIyBjb25zaWRlciBhY2NvdW50aW5nIGZvciBwcm92aXJ1cyBkZXNpZ25hdGlvbgojICAgIGtlZXBfc2NvcmVbKGlucHV0X3NlcXMkY2hlY2t2X3ZpcmFsX2dlbmVzKjMpIDw9IGlucHV0X3NlcXMkY2hlY2t2X2hvc3RfZ2VuZXNdIDwtIDAgIyBjb25zaWRlciBhY2NvdW50aW5nIGZvciBwcm92aXJ1cyBkZXNpZ25hdGlvbgogICAga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNoZWNrdl9sZW5ndGg+NTAwMDAwICYgaW5wdXRfc2VxcyRoYWxsbWFyaz09MF0gPC0ga2VlcF9zY29yZVtpbnB1dF9zZXFzJGNoZWNrdl9sZW5ndGg+NTAwMDAwICYgaW5wdXRfc2VxcyRoYWxsbWFyaz09MF0gLSAxCiAgfQogIAogIHJldHVybihrZWVwX3Njb3JlKQogIAp9CmBgYAoKIyBBc3Nlc3NpbmcgcGVyZm9ybWFuY2UgYWdhaW5zdCB0aGUgInRydXRoIgpub3RlIHRoYXQgdGhpcyBpcyBvbmx5IGFzIGFjY3VyYXRlIGFzIHRoZSBhbm5vdGF0aW9ucyBvZiB0aGUgaW5wdXQgc2VxdWVuY2VzCgp0aGlzIGZ1bmN0aW9uIGNhbGN1bGF0ZXMgdGhlIHByZWNpc2lvbiwgcmVjYWxsLCBhbmQgRjEgc2NvcmUgZm9yIGVhY2ggcGlwZWxpbmUKYGBge3J9CmFzc2Vzc19wZXJmb3JtYW5jZSA8LSBmdW5jdGlvbihzZXF0eXBlLCBrZWVwX3Njb3JlKSB7CiAgCiAgdHJ1ZXBvc2l0aXZlIDwtIHJlcCgibm90IHZpcmFsIiwgbGVuZ3RoKHNlcXR5cGUpKQogIHRydWVwb3NpdGl2ZVtzZXF0eXBlPT0idmlydXMiXSA8LSAidmlyYWwiCiAgCiAgI21ha2UgY29uZnVzaW9uIG1hdHJpeAogIGNvbmZ1c2lvbl9tYXRyaXggPC0gcmVwKCJ0cnVlIG5lZ2F0aXZlIiwgbGVuZ3RoKGtlZXBfc2NvcmUpKQogIGNvbmZ1c2lvbl9tYXRyaXhbdHJ1ZXBvc2l0aXZlPT0idmlyYWwiICYga2VlcF9zY29yZTw9MV0gPC0gImZhbHNlIG5lZ2F0aXZlIgogIGNvbmZ1c2lvbl9tYXRyaXhbdHJ1ZXBvc2l0aXZlPT0idmlyYWwiICYga2VlcF9zY29yZT49MV0gPC0gInRydWUgcG9zaXRpdmUiCiAgY29uZnVzaW9uX21hdHJpeFt0cnVlcG9zaXRpdmU9PSJub3QgdmlyYWwiICYga2VlcF9zY29yZT49MV0gPC0gImZhbHNlIHBvc2l0aXZlIgogIAogIFRQIDwtIHRhYmxlKGNvbmZ1c2lvbl9tYXRyaXgpWzRdCiAgRlAgPC0gdGFibGUoY29uZnVzaW9uX21hdHJpeClbMl0KICBUTiA8LSB0YWJsZShjb25mdXNpb25fbWF0cml4KVszXQogIEZOIDwtIHRhYmxlKGNvbmZ1c2lvbl9tYXRyaXgpWzFdCiAgCiAgcHJlY2lzaW9uIDwtIFRQLyhUUCtGUCkKICByZWNhbGwgPC0gVFAvKFRQK0ZOKQogIEYxIDwtIDIqcHJlY2lzaW9uKnJlY2FsbC8ocHJlY2lzaW9uK3JlY2FsbCkKICAKICBNQ0MgPC0gKFRQKlROLUZQKkZOKS9zcXJ0KGFzLm51bWVyaWMoVFArRlApKmFzLm51bWVyaWMoVFArRk4pKmFzLm51bWVyaWMoVE4rRlApKmFzLm51bWVyaWMoVE4rRk4pKQogIAogIGF1YyA8LSByb3VuZChhdWModHJ1ZXBvc2l0aXZlLCBrZWVwX3Njb3JlKSw0KQogIAogICNieSB0eXBlIG1ldHJpY3MKICBmdW5nYWxfRlAgPC0gdGFibGUoY29uZnVzaW9uX21hdHJpeFtzZXF0eXBlPT0iZnVuZ2kiXSlbMl0KICBwcm90aXN0X0ZQIDwtIHRhYmxlKGNvbmZ1c2lvbl9tYXRyaXhbc2VxdHlwZT09InByb3Rpc3QiXSlbMl0KICBiYWN0ZXJpYWxfRlAgPC0gdGFibGUoY29uZnVzaW9uX21hdHJpeFtzZXF0eXBlPT0iYmFjdGVyaWEiXSlbMl0KICB2aXJhbF9GTiA8LSB0YWJsZShjb25mdXNpb25fbWF0cml4W3NlcXR5cGU9PSJ2aXJ1cyJdKVsxXQogIAogIHBlcmZvcm1hbmNlIDwtIGMocHJlY2lzaW9uLCByZWNhbGwsIEYxLCBNQ0MsIGF1YywgZnVuZ2FsX0ZQLCAKICAgICAgICAgICAgICAgICAgIHByb3Rpc3RfRlAsIGJhY3RlcmlhbF9GUCwgdmlyYWxfRk4pCiAgbmFtZXMocGVyZm9ybWFuY2UpIDwtIGMoInByZWNpc2lvbiIsICJyZWNhbGwiLCAiRjEiLCAiTUNDIiwgIkFVQyIsICJmdW5nYWxfRlAiLAogICAgICAgICAgICAgICAgICAgICAgICAgICJwcm90aXN0X0ZQIiwgImJhY3RlcmlhbF9GUCIsICJ2aXJhbF9GTiIpCiAgCiAgcmV0dXJuKHBlcmZvcm1hbmNlKQp9CmBgYAoKY29tYmluYXRpb24gb2YgdG9vbHMgbGlzdApgYGB7cn0KY29tYm9zX2xpc3QgPC0gZGF0YS5mcmFtZSh0b29sY29tYm89cmVwKDAsIDY0KSwKICAgICAgICAgICAgICAgICAgICAgICAgICBDaGVja1Y9cmVwKDAsIDY0KSwKICAgICAgICAgICAgICAgICAgICAgICAgICBEVkY9cmVwKDAsIDY0KSwKICAgICAgICAgICAgICAgICAgICAgICAgICBLYWlqdT1yZXAoMCwgNjQpLAogICAgICAgICAgICAgICAgICAgICAgICAgIFZJQlJBTlQ9cmVwKDAsIDY0KSwKICAgICAgICAgICAgICAgICAgICAgICAgICBWUz1yZXAoMCwgNjQpLAogICAgICAgICAgICAgICAgICAgICAgICAgIFZTMj1yZXAoMCwgNjQpKQpwIDwtIDEKCmZvciAoaSBpbiBjKDAsMSkpewogIGZvciAoaiBpbiBjKDAsMSkpewogICAgZm9yIChrIGluIGMoMCwxKSl7CiAgICAgIGZvciAobCBpbiBjKDAsMSkpewogICAgICAgIGZvciAobSBpbiBjKDAsMSkpewogICAgICAgICAgZm9yIChuIGluIGMoMCwxKSl7CiAgICAgICAgICAgIGNvbWJvc19saXN0JHRvb2xjb21ib1twXSA8LSBwYXN0ZShpLGosayxsLG0sbikKICAgICAgICAgICAgY29tYm9zX2xpc3QkdG9vbGNvbWJvMltwXSA8LSBwYXN0ZShpZihpKXsiQyJ9ZWxzZXsiMCJ9LGlmKGopeyJEVkYifWVsc2V7IjAifSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZihrKXsiSyJ9ZWxzZXsiMCJ9LGlmKGwpeyJWQiJ9ZWxzZXsiMCJ9LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlmKG0peyJWUyJ9ZWxzZXsiMCJ9LGlmKG4peyJWUzIifWVsc2V7IjAifSkKICAgICAgICAgICAgY29tYm9zX2xpc3QkQ2hlY2tWW3BdIDwtIGkKICAgICAgICAgICAgY29tYm9zX2xpc3QkRFZGW3BdIDwtIGoKICAgICAgICAgICAgY29tYm9zX2xpc3QkS2FpanVbcF0gPC0gawogICAgICAgICAgICBjb21ib3NfbGlzdCRWSUJSQU5UW3BdIDwtIGwKICAgICAgICAgICAgY29tYm9zX2xpc3QkVlNbcF0gPC0gbQogICAgICAgICAgICBjb21ib3NfbGlzdCRWUzJbcF0gPC0gbgogICAgICAgICAgICBwIDwtIHArMQogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0KfQoKY29tYm9zX2xpc3QgPC0gY29tYm9zX2xpc3RbLTEsXQpgYGAKCnRoaXMgZnVuY3Rpb24gYnVpbGRzIGEgbGlzdCBvZiBhbGwgb2YgdGhlIGNvbWJpbmF0aW9ucyB0aGF0IHRoZSB1c2VyIHdhbnRzIHRvIAp0ZXN0LiAKSW4gdGhpcyBjYXNlLCB3ZSdyZSBjb21wYXJpbmcgdGhlIHBlcmZvcm1hbmNlIG9mIGFsbCB1bmlxdWUgY29tYmluYXRpb25zIG9mIHRoZSAKc2l4IHRvb2xzLgpgYGB7cn0KYnVpbGRfc2NvcmVfbGlzdCA8LSBmdW5jdGlvbihpbnB1dF9zZXFzLCBjb21ib3MpIHsKICBvdXRwdXQgPC0gZGF0YS5mcmFtZShwcmVjaXNpb249cmVwKDAsIG5yb3coY29tYm9zKSksCiAgICAgICAgICAgICAgICAgICAgICAgcmVjYWxsPXJlcCgwLCBucm93KGNvbWJvcykpLAogICAgICAgICAgICAgICAgICAgICAgIEYxPXJlcCgwLCBucm93KGNvbWJvcykpLAogICAgICAgICAgICAgICAgICAgICAgIE1DQz1yZXAoMCwgbnJvdyhjb21ib3MpKSwKICAgICAgICAgICAgICAgICAgICAgICBBVUM9cmVwKDAsIG5yb3coY29tYm9zKSksCiAgICAgICAgICAgICAgICAgICAgICAgZnVuZ2FsX0ZQPXJlcCgwLCBucm93KGNvbWJvcykpLAogICAgICAgICAgICAgICAgICAgICAgIHByb3Rpc3RfRlA9cmVwKDAsIG5yb3coY29tYm9zKSksCiAgICAgICAgICAgICAgICAgICAgICAgYmFjdGVyaWFsX0ZQPXJlcCgwLCBucm93KGNvbWJvcykpLAogICAgICAgICAgICAgICAgICAgICAgIHZpcmFsX0ZOPXJlcCgwLCBucm93KGNvbWJvcykpKQogIGZvciAoaSBpbiAxOm5yb3coY29tYm9zKSkgewogICAga2VlcF9zY29yZSA8LSBnZXR0aW5nX3ZpcmFsX3NldF8xKGlucHV0X3NlcXMsIGluY2x1ZGVfdmlicmFudCA9IGNvbWJvcyRWSUJSQU5UW2ldLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyID0gY29tYm9zJFZTW2ldLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyMiA9IGNvbWJvcyRWUzJbaV0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV90dW5pbmcgPSBjb21ib3MkQ2hlY2tWW2ldLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfa2FpanUgPSBjb21ib3MkS2FpanVbaV0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV9kZWVwdmlyZmluZGVyID0gY29tYm9zJERWRltpXSkKICAKICAgIG91dHB1dFtpLDE6OV0gPC0gYXNzZXNzX3BlcmZvcm1hbmNlKGlucHV0X3NlcXMkc2VxdHlwZSwga2VlcF9zY29yZSkKICAgIAogICAgb3V0cHV0JHRvb2xjb21ib1tpXSA8LSBwYXN0ZShjb21ib3MkQ2hlY2tWW2ldLGNvbWJvcyREVkZbaV0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbWJvcyRLYWlqdVtpXSwgY29tYm9zJFZJQlJBTlRbaV0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbWJvcyRWU1tpXSwgY29tYm9zJFZTMltpXSkKICB9CiAgCiAgb3V0cHV0W2lzLm5hKG91dHB1dCldIDwtIDAKCiAgcmV0dXJuIChvdXRwdXQpCn0KYGBgCgojIyBDYWxjdWxhdGUgdGhlIHBlcmZvcm1hbmNlIG9mIGVhY2ggcGlwZWxpbmUKYGBge3J9CmFjY3VyYWN5X3Njb3JlcyA8LSBkYXRhLmZyYW1lKHRlc3Rpbmdfc2V0X2luZGV4PXJlcCgwLCBucm93KGNvbWJvc19saXN0KSoxMCksCiAgICAgICAgICAgICAgICAgICAgICBwcmVjaXNpb249cmVwKDAsIG5yb3coY29tYm9zX2xpc3QpKjEwKSwKICAgICAgICAgICAgICAgICAgICAgICByZWNhbGw9cmVwKDAsIG5yb3coY29tYm9zX2xpc3QpKjEwKSwKICAgICAgICAgICAgICAgICAgICAgICBGMT1yZXAoMCwgbnJvdyhjb21ib3NfbGlzdCkqMTApLAogICAgICAgICAgICAgICAgICAgICAgIE1DQz1yZXAoMCwgbnJvdyhjb21ib3NfbGlzdCkqMTApLCAKICAgICAgICAgICAgICAgICAgICAgIEFVQz1yZXAoMCwgbnJvdyhjb21ib3NfbGlzdCkqMTApLAogICAgICAgICAgICAgICAgICAgICAgZnVuZ2FsX0ZQPXJlcCgwLCBucm93KGNvbWJvc19saXN0KSoxMCksCiAgICAgICAgICAgICAgICAgICAgICBwcm90aXN0X0ZQPXJlcCgwLCBucm93KGNvbWJvc19saXN0KSoxMCksCiAgICAgICAgICAgICAgICAgICAgICBiYWN0ZXJpYWxfRlA9cmVwKDAsIG5yb3coY29tYm9zX2xpc3QpKjEwKSwKICAgICAgICAgICAgICAgICAgICAgIHZpcmFsX0ZOPXJlcCgwLCBucm93KGNvbWJvc19saXN0KSoxMCkpCgphY2N1cmFjeV9zY29yZXMgPC0gY2JpbmQodGVzdGluZ19zZXRfaW5kZXg9cmVwKDEsIG5yb3coY29tYm9zX2xpc3QpKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYnVpbGRfc2NvcmVfbGlzdCh2aXJ1c2VzW3ZpcnVzZXMkSW5kZXg9PTEsXSwgY29tYm9zX2xpc3QpKQpmb3IgKGkgaW4gMjoxMCkgewogIGFjY3VyYWN5X3Njb3JlcyA8LSByYmluZChhY2N1cmFjeV9zY29yZXMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgIGNiaW5kKHRlc3Rpbmdfc2V0X2luZGV4PXJlcChpLCBucm93KGNvbWJvc19saXN0KSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJ1aWxkX3Njb3JlX2xpc3QodmlydXNlc1t2aXJ1c2VzJEluZGV4PT1pLF0sIGNvbWJvc19saXN0KSkpCn0KYGBgCgpgYGB7cn0KbGlicmFyeSgic3RyaW5nciIpCmBgYAoKYGBge3J9CmFjY3VyYWN5X3Njb3JlcyRudW10b29scyA8LSBzdHJfY291bnQoYWNjdXJhY3lfc2NvcmVzJHRvb2xjb21ibywgIjEiKQojYWNjdXJhY3lfc2NvcmVzIDwtIGFjY3VyYWN5X3Njb3Jlc1tvcmRlcihhY2N1cmFjeV9zY29yZXMkbnVtdG9vbHMsIGRlY3JlYXNpbmc9RiksXQphY2N1cmFjeV9zY29yZXMgPC0gYWNjdXJhY3lfc2NvcmVzW29yZGVyKGFjY3VyYWN5X3Njb3JlcyRNQ0MsIGRlY3JlYXNpbmc9RiksXQphY2N1cmFjeV9zY29yZXMkdG9vbGNvbWJvIDwtIGZhY3RvcihhY2N1cmFjeV9zY29yZXMkdG9vbGNvbWJvLCBsZXZlbHMgPSB1bmlxdWUoYWNjdXJhY3lfc2NvcmVzJHRvb2xjb21ibykpCmFjY3VyYWN5X3Njb3JlcyRudW10b29scyA8LSBhcy5mYWN0b3IoYWNjdXJhY3lfc2NvcmVzJG51bXRvb2xzKQpgYGAKCgojIyBWaXN1YWxpemUgaG93IHRoZSBwcmVjaXNpb24sIHJlY2FsbCwgYW5kIEYxIHNjb3JlcyBjaGFuZ2UgYWNyb3NzIHBpcGVsaW5lcy4KYGBge3J9CnBhbCA8LSBnZ3RoZW1lczo6dGFibGVhdV9jb2xvcl9wYWwocGFsZXR0ZT0iVGFibGVhdSAxMCIsIHR5cGU9InJlZ3VsYXIiKQpwMiA8LSBnZ3Bsb3QoYWNjdXJhY3lfc2NvcmVzLCBhZXMoeD10b29sY29tYm8sIHk9RjEsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9bnVtdG9vbHMsIGZpbGw9bnVtdG9vbHMpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjUpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQsIGFuZ2xlID0gOTApLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiVG9vbCBDb21iaW5hdGlvbiAoQ1YsIERWRiwgS0osIFZCLCBWUywgVlMyKSIpICsKICB5bGFiKCJGMSBTY29yZSIpCnAyCmdncGxvdChhY2N1cmFjeV9zY29yZXMsIGFlcyh4PXRvb2xjb21ibywgeT1wcmVjaXNpb24sIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9bnVtdG9vbHMsIGZpbGw9bnVtdG9vbHMpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjUpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQsIGFuZ2xlID0gOTApLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiVG9vbCBDb21iaW5hdGlvbiAoQ1YsIERWRiwgS0osIFZCLCBWUywgVlMyKSIpICsKICB5bGFiKCJQcmVjaXNpb24iKQpnZ3Bsb3QoYWNjdXJhY3lfc2NvcmVzLCBhZXMoeD10b29sY29tYm8sIHk9cmVjYWxsLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yPW51bXRvb2xzLCBmaWxsPW51bXRvb2xzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC41KSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0LCBhbmdsZSA9IDkwKSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIlRvb2wgQ29tYmluYXRpb24gKENWLCBEVkYsIEtKLCBWQiwgVlMsIFZTMikiKSArCiAgeWxhYigiUmVjYWxsIikKZ2dwbG90KGFjY3VyYWN5X3Njb3JlcywgYWVzKHg9cHJlY2lzaW9uLCB5PXJlY2FsbCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xvcj1udW10b29scywgZmlsbD1udW10b29scykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuNSkgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCwgYW5nbGUgPSA5MCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJQcmVjaXNpb24iKSArCiAgeWxhYigiUmVjYWxsIikKZ2dwbG90KGFjY3VyYWN5X3Njb3JlcywgYWVzKHg9dG9vbGNvbWJvLCB5PWFicyhwcmVjaXNpb24tcmVjYWxsKSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xvcj1udW10b29scywgZmlsbD1udW10b29scykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuNSkgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCwgYW5nbGUgPSA5MCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJUb29sIENvbWJpbmF0aW9uIChDViwgRFZGLCBLSiwgVkIsIFZTLCBWUzIpIikgKwogIHlsYWIoIlByZWNpc2lvbi1SZWNhbGwiKQpnZ3Bsb3QoYWNjdXJhY3lfc2NvcmVzLCBhZXMoeD10b29sY29tYm8sIHk9TUNDLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yPW51bXRvb2xzLCBmaWxsPW51bXRvb2xzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC41KSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0LCBhbmdsZSA9IDkwKSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIlRvb2wgQ29tYmluYXRpb24gKENWLCBEVkYsIEtKLCBWQiwgVlMsIFZTMikiKSArCiAgeWxhYigiTUNDIikKZ2dwbG90KGFjY3VyYWN5X3Njb3JlcywgYWVzKHg9dG9vbGNvbWJvLCB5PUFVQywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xvcj1udW10b29scywgZmlsbD1udW10b29scykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuNSkgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCwgYW5nbGUgPSA5MCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJUb29sIENvbWJpbmF0aW9uIChDViwgRFZGLCBLSiwgVkIsIFZTLCBWUzIpIikgKwogIHlsYWIoIkFVQyIpCmdncGxvdChhY2N1cmFjeV9zY29yZXMsIGFlcyh4PXRvb2xjb21ibywgeT1mdW5nYWxfRlAsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9bnVtdG9vbHMsIGZpbGw9bnVtdG9vbHMpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjUpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQsIGFuZ2xlID0gOTApLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiVG9vbCBDb21iaW5hdGlvbiAoQ1YsIERWRiwgS0osIFZCLCBWUywgVlMyKSIpICsKICB5bGFiKCJGdW5nYWwgRmFsc2UgUG9zaXRpdmVzIikKCmdncGxvdChhY2N1cmFjeV9zY29yZXMsIGFlcyh4PXRvb2xjb21ibywgeT1wcm90aXN0X0ZQLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yPW51bXRvb2xzLCBmaWxsPW51bXRvb2xzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC41KSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0LCBhbmdsZSA9IDkwKSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIlRvb2wgQ29tYmluYXRpb24gKENWLCBEVkYsIEtKLCBWQiwgVlMsIFZTMikiKSArCiAgeWxhYigiUHJvdGlzdCBGYWxzZSBQb3NpdGl2ZXMiKQoKZ2dwbG90KGFjY3VyYWN5X3Njb3JlcywgYWVzKHg9dG9vbGNvbWJvLCB5PWJhY3RlcmlhbF9GUCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xvcj1udW10b29scywgZmlsbD1udW10b29scykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuNSkgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCwgYW5nbGUgPSA5MCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJUb29sIENvbWJpbmF0aW9uIChDViwgRFZGLCBLSiwgVkIsIFZTLCBWUzIpIikgKwogIHlsYWIoIkJhY3RlcmlhbCBGYWxzZSBQb3NpdGl2ZXMiKQpnZ3Bsb3QoYWNjdXJhY3lfc2NvcmVzLCBhZXMoeD10b29sY29tYm8sIHk9dmlyYWxfRk4sIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9bnVtdG9vbHMsIGZpbGw9bnVtdG9vbHMpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjUpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQsIGFuZ2xlID0gOTApLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiVG9vbCBDb21iaW5hdGlvbiAoQ1YsIERWRiwgS0osIFZCLCBWUywgVlMyKSIpICsKICB5bGFiKCJWaXJhbCBGYWxzZSBOZWdhdGl2ZXMiKQpgYGAKCmBgYHtyfQp3cml0ZV90c3YoYWNjdXJhY3lfc2NvcmVzLCAiMjAyMjA5MjdfYWNjdXJhY3lfc2NvcmVzLnRzdiIpCmBgYAoKdG8gZG86IGFkZCBpbiBjbHVzdGVyaW5nIGFuZCBvcmRpbmF0aW9uIGxpa2UgaW4gdGhlIGRyaW5raW5nIHdhdGVyIFIgbm90ZWJvb2sKCiMgRXhwZXJpbWVudGluZwoKIyMgaGlnaCBwcmVjaXNpb24gZXhhbXBsZQpgYGB7cn0KdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfcHJlY2lzaW9uIDwtIGdldHRpbmdfdmlyYWxfc2V0XzEodmlydXNlcywgaW5jbHVkZV9kZWVwdmlyZmluZGVyID0gRiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlicmFudCA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlcjIgPSBGLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV9rYWlqdSA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3R1bmluZyA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlciA9IEYpCmBgYAoKCmBgYHtyfQp2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24gPC0gInRydWUgbmVnYXRpdmUiCnZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX3ByZWNpc2lvblt2aXJ1c2VzJHNlcXR5cGU9PSJ2aXJ1cyIgJiB2aXJ1c2VzJGtlZXBfc2NvcmVfaGlnaF9wcmVjaXNpb248MV0gPC0gImZhbHNlIG5lZ2F0aXZlIgp2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb25bdmlydXNlcyRzZXF0eXBlPT0idmlydXMiICYgdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfcHJlY2lzaW9uPj0xXSA8LSAidHJ1ZSBwb3NpdGl2ZSIKdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uW3ZpcnVzZXMkc2VxdHlwZSE9InZpcnVzIiAmIHZpcnVzZXMka2VlcF9zY29yZV9oaWdoX3ByZWNpc2lvbj49MV0gPC0gImZhbHNlIHBvc2l0aXZlIgpgYGAKCnZpc3VhbGl6aW5nIGNvbmZ1c2lvbiBtYXRyaXggYnkgdGF4YQpgYGB7cn0KY29uZnVzaW9uX2J5X3RheGEgPC0gbWVsdCh0YWJsZSh2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24sIHZpcnVzZXMkc2VxdHlwZSwgdmlydXNlcyRJbmRleCkpCmNvbG5hbWVzKGNvbmZ1c2lvbl9ieV90YXhhKSA8LSBjKCJjb25mdXNpb25fbWF0cml4IiwgInNlcXR5cGUiLCJJbmRleCIsICJjb3VudCIpCmBgYAoKCgpgYGB7cn0KcGFsIDwtIGdndGhlbWVzOjp0YWJsZWF1X2NvbG9yX3BhbChwYWxldHRlPSJUYWJsZWF1IDEwIiwgdHlwZT0icmVndWxhciIpCmBgYAoKYGBge3J9CmdncGxvdChjb25mdXNpb25fYnlfdGF4YSwgYWVzKHg9Y291bnQsIHk9YXMuZmFjdG9yKEluZGV4KSwKICAgICAgICAgICAgICAgICAgIGZpbGw9Y29uZnVzaW9uX21hdHJpeCwKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNvbmZ1c2lvbl9tYXRyaXgpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHNjYWxlX2ZpbGxfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAwLjUpLAogICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJmYWxzZSBuZWdhdGl2ZSIsICJmYWxzZSBwb3NpdGl2ZSIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ0cnVlIG5lZ2F0aXZlIiwgInRydWUgcG9zaXRpdmUiKSkgKwogIHNjYWxlX2NvbG9yX21hbnVhbChuYW1lPSIiLAogICAgICAgICAgICAgICAgICAgICB2YWx1ZXMgPSBhbHBoYShyZXYocGFsKDQpKSwgMSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgeGxhYigiTnVtYmVyIG9mIFNlcXVlbmNlcyIpICsKICB5bGFiKCIiKSArIAogIGZhY2V0X3dyYXAofnNlcXR5cGUsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQpgYGAKCmBgYHtyfQogZ2dwbG90KHZpcnVzZXMsIGFlcyh4PWNoZWNrdl92aXJhbF9nZW5lcywgeT1jb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uLAogICAgICAgICAgICAgICAgICAgZmlsbD1jb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uLAogICAgICAgICAgICAgICAgICAgY29sb3I9Y29uZnVzaW9uX21hdHJpeF9oaWdoX3ByZWNpc2lvbikpICsKICBnZW9tX2JveHBsb3QoYWxwaGE9MC4zKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHNjYWxlX2ZpbGxfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAwLjUpLAogICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJmYWxzZSBuZWdhdGl2ZSIsICJmYWxzZSBwb3NpdGl2ZSIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ0cnVlIG5lZ2F0aXZlIiwgInRydWUgcG9zaXRpdmUiKSkgKwogIHNjYWxlX2NvbG9yX21hbnVhbChuYW1lPSIiLAogICAgICAgICAgICAgICAgICAgICB2YWx1ZXMgPSBhbHBoYShyZXYocGFsKDQpKSwgMSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgeGxhYigiTnVtYmVyIG9mIFZpcmFsIFNlcXVlbmNlcyIpICsKICB5bGFiKCIiKSArIAogIGZhY2V0X3dyYXAofnNlcXR5cGUsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQoKIGdncGxvdCh2aXJ1c2VzLCBhZXMoeD1wZXJjZW50X3ZpcmFsLCB5PWNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBmaWxsPWNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBjb2xvcj1jb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uKSkgKwogIGdlb21fYm94cGxvdChhbHBoYT0wLjMpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgc2NhbGVfZmlsbF9tYW51YWwobmFtZT0iIiwKICAgICAgICAgICAgICAgICAgICAgdmFsdWVzID0gYWxwaGEocmV2KHBhbCg0KSksIDAuNSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgc2NhbGVfY29sb3JfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAxKSwKICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiZmFsc2UgbmVnYXRpdmUiLCAiZmFsc2UgcG9zaXRpdmUiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidHJ1ZSBuZWdhdGl2ZSIsICJ0cnVlIHBvc2l0aXZlIikpICsKICB4bGFiKCJQZXJjZW50IEdlbmVzIFZpcmFsIikgKwogIHlsYWIoIiIpICsgCiAgZmFjZXRfd3JhcCh+c2VxdHlwZSwgc2NhbGVzID0gImZyZWUiKSArCiAgY29vcmRfZmxpcCgpCgogZ2dwbG90KHZpcnVzZXMsIGFlcyh4PWhhbGxtYXJrLCB5PWNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBmaWxsPWNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBjb2xvcj1jb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uKSkgKwogIGdlb21fYm94cGxvdChhbHBoYT0wLjMpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgc2NhbGVfZmlsbF9tYW51YWwobmFtZT0iIiwKICAgICAgICAgICAgICAgICAgICAgdmFsdWVzID0gYWxwaGEocmV2KHBhbCg0KSksIDAuNSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgc2NhbGVfY29sb3JfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAxKSwKICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiZmFsc2UgbmVnYXRpdmUiLCAiZmFsc2UgcG9zaXRpdmUiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidHJ1ZSBuZWdhdGl2ZSIsICJ0cnVlIHBvc2l0aXZlIikpICsKICB4bGFiKCJOdW1iZXIgb2YgSGFsbG1hcmsgR2VuZXMiKSArCiAgeWxhYigiIikgKyAKICBmYWNldF93cmFwKH5zZXF0eXBlLCBzY2FsZXMgPSAiZnJlZSIpICsKICBjb29yZF9mbGlwKCkKIApnZ3Bsb3QodmlydXNlcywgYWVzKHg9aGFsbG1hcmssIHk9Y2hlY2t2X3ZpcmFsX2dlbmVzLAogICAgICAgICAgICAgICAgICAgZmlsbD1jb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uLAogICAgICAgICAgICAgICAgICAgY29sb3I9Y29uZnVzaW9uX21hdHJpeF9oaWdoX3ByZWNpc2lvbikpICsKICBnZW9tX3BvaW50KGFscGhhPTAuMykgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICBzY2FsZV9maWxsX21hbnVhbChuYW1lPSIiLAogICAgICAgICAgICAgICAgICAgICB2YWx1ZXMgPSBhbHBoYShyZXYocGFsKDQpKSwgMC41KSwKICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiZmFsc2UgbmVnYXRpdmUiLCAiZmFsc2UgcG9zaXRpdmUiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidHJ1ZSBuZWdhdGl2ZSIsICJ0cnVlIHBvc2l0aXZlIikpICsKICBzY2FsZV9jb2xvcl9tYW51YWwobmFtZT0iIiwKICAgICAgICAgICAgICAgICAgICAgdmFsdWVzID0gYWxwaGEocmV2KHBhbCg0KSksIDEpLAogICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJmYWxzZSBuZWdhdGl2ZSIsICJmYWxzZSBwb3NpdGl2ZSIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ0cnVlIG5lZ2F0aXZlIiwgInRydWUgcG9zaXRpdmUiKSkgKwogIHhsYWIoIk51bWJlciBvZiBIYWxsbWFyayBHZW5lcyIpICsKICB5bGFiKCJOdW1iZXIgb2YgVmlyYWwgR2VuZXMiKSArIAogIGZhY2V0X3dyYXAofnNlcXR5cGUsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQpgYGAKCmBgYHtyfQp2aXJ1c2VzX2ZhbHNlX3Bvc2l0aXZlIDwtIHZpcnVzZXNbdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uPT0iZmFsc2UgcG9zaXRpdmUiLF0KdmlydXNlc19mYWxzZV9uZWdhdGl2ZSA8LSB2aXJ1c2VzW3ZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX3ByZWNpc2lvbj09ImZhbHNlIG5lZ2F0aXZlIixdCmBgYAoKYGBge3J9CmdncGxvdCh2aXJ1c2VzLCBhZXMoeD1oYWxsbWFyaywgeT1jaGVja3ZfdmlyYWxfZ2VuZXMsCiAgICAgICAgICAgICAgICAgICBmaWxsPWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBjb2xvcj1jaGVja3ZfbGVuZ3RoLAogICAgICAgICAgICAgICAgICAgc2hhcGU9Y2hlY2t2X3Byb3ZpcnVzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC4zKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIk51bWJlciBvZiBIYWxsbWFyayBHZW5lcyIpICsKICB5bGFiKCJOdW1iZXIgb2YgVmlyYWwgR2VuZXMiKSArIAogIGZhY2V0X3dyYXAofnNlcXR5cGUsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQoKZ2dwbG90KHZpcnVzZXNfZmFsc2VfcG9zaXRpdmUsIGFlcyh4PWhhbGxtYXJrLCB5PWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBmaWxsPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIHNoYXBlPWNoZWNrdl9wcm92aXJ1cykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuMykgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJOdW1iZXIgb2YgSGFsbG1hcmsgR2VuZXMiKSArCiAgeWxhYigiQ29udGlnIExlbmd0aCIpICsgCiAgZmFjZXRfd3JhcCh+c2VxdHlwZSwgc2NhbGVzID0gImZyZWUiKSArCiAgY29vcmRfZmxpcCgpCgpnZ3Bsb3QodmlydXNlc19mYWxzZV9wb3NpdGl2ZVt2aXJ1c2VzX2ZhbHNlX3Bvc2l0aXZlJHNlcXR5cGU9PSJiYWN0ZXJpYSJdLCBhZXMoeD1oYWxsbWFyaywgeT1jaGVja3ZfbGVuZ3RoLAogICAgICAgICAgICAgICAgICAgZmlsbD1jaGVja3ZfdmlyYWxfZ2VuZXMsCiAgICAgICAgICAgICAgICAgICBjb2xvcj1jaGVja3ZfdmlyYWxfZ2VuZXMsCiAgICAgICAgICAgICAgICAgICBzaGFwZT1jaGVja3ZfcHJvdmlydXMpKSArCiAgZ2VvbV9wb2ludChhbHBoYT0wLjMpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiTnVtYmVyIG9mIEhhbGxtYXJrIEdlbmVzIikgKwogIHlsYWIoIkNvbnRpZyBMZW5ndGgiKSArIAogIGZhY2V0X3dyYXAofkthaWp1X1ZpcmFsLCBzY2FsZXMgPSAiZnJlZSIpICsKICBjb29yZF9mbGlwKCkKCmdncGxvdCh2aXJ1c2VzX2ZhbHNlX3Bvc2l0aXZlW3ZpcnVzZXNfZmFsc2VfcG9zaXRpdmUkc2VxdHlwZT09ImZ1bmdpIl0sIGFlcyh4PWhhbGxtYXJrLCB5PWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBmaWxsPWtlZXBfc2NvcmVfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBjb2xvcj1rZWVwX3Njb3JlX2hpZ2hfcHJlY2lzaW9uLAogICAgICAgICAgICAgICAgICAgc2hhcGU9Y2hlY2t2X3Byb3ZpcnVzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC4zKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIk51bWJlciBvZiBIYWxsbWFyayBHZW5lcyIpICsKICB5bGFiKCJDb250aWcgTGVuZ3RoIikgKyAKICBmYWNldF93cmFwKH5LYWlqdV9WaXJhbCwgc2NhbGVzID0gImZyZWUiKSArCiAgY29vcmRfZmxpcCgpCgpnZ3Bsb3QodmlydXNlc19mYWxzZV9wb3NpdGl2ZVt2aXJ1c2VzX2ZhbHNlX3Bvc2l0aXZlJHNlcXR5cGU9PSJwcm90aXN0Il0sIGFlcyh4PWhhbGxtYXJrLCB5PWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBmaWxsPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIHNoYXBlPWNoZWNrdl9wcm92aXJ1cykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuMykgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJOdW1iZXIgb2YgSGFsbG1hcmsgR2VuZXMiKSArCiAgeWxhYigiQ29udGlnIExlbmd0aCIpICsgCiAgZmFjZXRfd3JhcCh+S2FpanVfVmlyYWwsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQoKZ2dwbG90KHZpcnVzZXNfZmFsc2VfbmVnYXRpdmUsIGFlcyh4PWhhbGxtYXJrLCB5PWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBmaWxsPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNoZWNrdl92aXJhbF9nZW5lcywKICAgICAgICAgICAgICAgICAgIHNoYXBlPWNoZWNrdl9wcm92aXJ1cykpICsKICBnZW9tX3BvaW50KGFscGhhPTAuMykgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJOdW1iZXIgb2YgSGFsbG1hcmsgR2VuZXMiKSArCiAgeWxhYigiQ29udGlnIExlbmd0aCIpICsgCiAgZmFjZXRfd3JhcCh+S2FpanVfVmlyYWwsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQoKZ2dwbG90KHZpcnVzZXNfZmFsc2VfbmVnYXRpdmUsIGFlcyh4PWhhbGxtYXJrLCB5PWNoZWNrdl9sZW5ndGgsCiAgICAgICAgICAgICAgICAgICBmaWxsPWtlZXBfc2NvcmVfaGlnaF9wcmVjaXNpb24sCiAgICAgICAgICAgICAgICAgICBjb2xvcj1rZWVwX3Njb3JlX2hpZ2hfcHJlY2lzaW9uLAogICAgICAgICAgICAgICAgICAgc2hhcGU9Y2hlY2t2X3Byb3ZpcnVzKSkgKwogIGdlb21fcG9pbnQoYWxwaGE9MC4zKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIk51bWJlciBvZiBIYWxsbWFyayBHZW5lcyIpICsKICB5bGFiKCJDb250aWcgTGVuZ3RoIikgKyAKICBmYWNldF93cmFwKH5LYWlqdV9WaXJhbCwgc2NhbGVzID0gImZyZWUiKSArCiAgY29vcmRfZmxpcCgpCmBgYAoKCgoKYGBge3J9CnRhYmxlKHZpcnVzZXMkaGFsbG1hcmtbdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uPT0iZmFsc2UgcG9zaXRpdmUiXT4wKQoKdGFibGUodmlydXNlcyRwZXJjZW50X2hvc3RbdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfcHJlY2lzaW9uPT0iZmFsc2UgcG9zaXRpdmUiXTw1MCkKYGBgCgojIyBoaWdoIE1DQyBleGFtcGxlCmBgYHtyfQp2aXJ1c2VzJGtlZXBfc2NvcmVfaGlnaF9NQ0MgPC0gZ2V0dGluZ192aXJhbF9zZXRfMSh2aXJ1c2VzLCBpbmNsdWRlX2RlZXB2aXJmaW5kZXIgPSBGLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV92aWJyYW50ID0gVCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyMiA9IEYsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX2thaWp1ID0gVCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdHVuaW5nID0gVCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyID0gVCkKYGBgCgoKYGBge3J9CnZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX01DQyA8LSAidHJ1ZSBuZWdhdGl2ZSIKdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfTUNDW3ZpcnVzZXMkc2VxdHlwZT09InZpcnVzIiAmIHZpcnVzZXMka2VlcF9zY29yZV9oaWdoX01DQzwxXSA8LSAiZmFsc2UgbmVnYXRpdmUiCnZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX01DQ1t2aXJ1c2VzJHNlcXR5cGU9PSJ2aXJ1cyIgJiB2aXJ1c2VzJGtlZXBfc2NvcmVfaGlnaF9NQ0M+PTFdIDwtICJ0cnVlIHBvc2l0aXZlIgp2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9NQ0NbdmlydXNlcyRzZXF0eXBlIT0idmlydXMiICYgdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfTUNDPj0xXSA8LSAiZmFsc2UgcG9zaXRpdmUiCmBgYAoKdmlzdWFsaXppbmcgY29uZnVzaW9uIG1hdHJpeCBieSB0YXhhCmBgYHtyfQpjb25mdXNpb25fYnlfdGF4YSA8LSBtZWx0KHRhYmxlKHZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX01DQywgdmlydXNlcyRzZXF0eXBlLCB2aXJ1c2VzJEluZGV4KSkKY29sbmFtZXMoY29uZnVzaW9uX2J5X3RheGEpIDwtIGMoImNvbmZ1c2lvbl9tYXRyaXgiLCAic2VxdHlwZSIsIkluZGV4IiwgImNvdW50IikKYGBgCgoKCmBgYHtyfQpwYWwgPC0gZ2d0aGVtZXM6OnRhYmxlYXVfY29sb3JfcGFsKHBhbGV0dGU9IlRhYmxlYXUgMTAiLCB0eXBlPSJyZWd1bGFyIikKYGBgCgpgYGB7cn0KZ2dwbG90KGNvbmZ1c2lvbl9ieV90YXhhLCBhZXMoeD1jb3VudCwgeT1hcy5mYWN0b3IoSW5kZXgpLAogICAgICAgICAgICAgICAgICAgZmlsbD1jb25mdXNpb25fbWF0cml4LAogICAgICAgICAgICAgICAgICAgY29sb3I9Y29uZnVzaW9uX21hdHJpeCkpICsKICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgc2NhbGVfZmlsbF9tYW51YWwobmFtZT0iIiwKICAgICAgICAgICAgICAgICAgICAgdmFsdWVzID0gYWxwaGEocmV2KHBhbCg0KSksIDAuNSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgc2NhbGVfY29sb3JfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAxKSwKICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiZmFsc2UgbmVnYXRpdmUiLCAiZmFsc2UgcG9zaXRpdmUiLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidHJ1ZSBuZWdhdGl2ZSIsICJ0cnVlIHBvc2l0aXZlIikpICsKICB4bGFiKCJOdW1iZXIgb2YgU2VxdWVuY2VzIikgKwogIHlsYWIoIiIpICsgCiAgZmFjZXRfd3JhcCh+c2VxdHlwZSwgc2NhbGVzID0gImZyZWUiKSArCiAgY29vcmRfZmxpcCgpCmBgYAoKIyMgaGlnaCByZWNhbGwgZXhhbXBsZQpgYGB7cn0KdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfcmVjYWxsIDwtIGdldHRpbmdfdmlyYWxfc2V0XzEodmlydXNlcywgaW5jbHVkZV9kZWVwdmlyZmluZGVyID0gVCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlicmFudCA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlcjIgPSBULAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV9rYWlqdSA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3R1bmluZyA9IFQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlciA9IFQpCmBgYAoKCmBgYHtyfQp2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9yZWNhbGwgPC0gInRydWUgbmVnYXRpdmUiCnZpcnVzZXMkY29uZnVzaW9uX21hdHJpeF9oaWdoX3JlY2FsbFt2aXJ1c2VzJHNlcXR5cGU9PSJ2aXJ1cyIgJiB2aXJ1c2VzJGtlZXBfc2NvcmVfaGlnaF9yZWNhbGw8MV0gPC0gImZhbHNlIG5lZ2F0aXZlIgp2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9yZWNhbGxbdmlydXNlcyRzZXF0eXBlPT0idmlydXMiICYgdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfcmVjYWxsPj0xXSA8LSAidHJ1ZSBwb3NpdGl2ZSIKdmlydXNlcyRjb25mdXNpb25fbWF0cml4X2hpZ2hfcmVjYWxsW3ZpcnVzZXMkc2VxdHlwZSE9InZpcnVzIiAmIHZpcnVzZXMka2VlcF9zY29yZV9oaWdoX3JlY2FsbD49MV0gPC0gImZhbHNlIHBvc2l0aXZlIgpgYGAKCnZpc3VhbGl6aW5nIGNvbmZ1c2lvbiBtYXRyaXggYnkgdGF4YQpgYGB7cn0KY29uZnVzaW9uX2J5X3RheGEgPC0gbWVsdCh0YWJsZSh2aXJ1c2VzJGNvbmZ1c2lvbl9tYXRyaXhfaGlnaF9yZWNhbGwsIHZpcnVzZXMkc2VxdHlwZSwgdmlydXNlcyRJbmRleCkpCmNvbG5hbWVzKGNvbmZ1c2lvbl9ieV90YXhhKSA8LSBjKCJjb25mdXNpb25fbWF0cml4IiwgInNlcXR5cGUiLCJJbmRleCIsICJjb3VudCIpCmBgYAoKCgpgYGB7cn0KcGFsIDwtIGdndGhlbWVzOjp0YWJsZWF1X2NvbG9yX3BhbChwYWxldHRlPSJUYWJsZWF1IDEwIiwgdHlwZT0icmVndWxhciIpCmBgYAoKYGBge3J9CnAyIDwtIGdncGxvdChjb25mdXNpb25fYnlfdGF4YSwgYWVzKHg9Y291bnQsIHk9YXMuZmFjdG9yKEluZGV4KSwKICAgICAgICAgICAgICAgICAgIGZpbGw9Y29uZnVzaW9uX21hdHJpeCwKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNvbmZ1c2lvbl9tYXRyaXgpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHNjYWxlX2ZpbGxfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNCkpLCAwLjUpLAogICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJmYWxzZSBuZWdhdGl2ZSIsICJmYWxzZSBwb3NpdGl2ZSIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ0cnVlIG5lZ2F0aXZlIiwgInRydWUgcG9zaXRpdmUiKSkgKwogIHNjYWxlX2NvbG9yX21hbnVhbChuYW1lPSIiLAogICAgICAgICAgICAgICAgICAgICB2YWx1ZXMgPSBhbHBoYShyZXYocGFsKDQpKSwgMSksCiAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoImZhbHNlIG5lZ2F0aXZlIiwgImZhbHNlIHBvc2l0aXZlIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInRydWUgbmVnYXRpdmUiLCAidHJ1ZSBwb3NpdGl2ZSIpKSArCiAgeGxhYigiTnVtYmVyIG9mIFNlcXVlbmNlcyIpICsKICB5bGFiKCIiKSArIAogIGZhY2V0X3dyYXAofnNlcXR5cGUsIHNjYWxlcyA9ICJmcmVlIikgKwogIGNvb3JkX2ZsaXAoKQpwMgpgYGAKCmxvb2tpbmcgYXQgZmFsc2UgbmVnYXRpdmVzCmBgYHtyfQp2aXJ1c2VzX2ZhbHNlX25lZ3MgPC0gdmlydXNlc1sodmlydXNlcyRzZXF0eXBlPT0idmlydXMiICYgdmlydXNlcyRrZWVwX3Njb3JlX2hpZ2hfcmVjYWxsPD0xKSxdCmBgYAoKCgoKIyBWaXN1YWxpemluZyBjb25mdXNpb24gbWF0cml4IGJ5IG51bWJlciBvZiB0b29scwoKCmBgYHtyfQp2aXJ1c2VzJGtlZXBfc2NvcmVfdmlzdWFsaXplIDwtIHZpcnVzZXMka2VlcF9zY29yZQp2aXJ1c2VzJGtlZXBfc2NvcmVfdmlzdWFsaXplW3ZpcnVzZXMka2VlcF9zY29yZT4xXSA8LSAiPjEiCnZpcnVzZXMka2VlcF9zY29yZV92aXN1YWxpemUgPC0gZmFjdG9yKHZpcnVzZXMka2VlcF9zY29yZV92aXN1YWxpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHM9YygiLTAuNSIsICItMSIsICIwIiwgIjAuNSIsIjEiLCAiPjEiKSkKdmlydXNlcyRrZWVwX3Njb3JlX3Zpc3VhbGl6ZSA8LSBmYWN0b3IodmlydXNlcyRrZWVwX3Njb3JlX3Zpc3VhbGl6ZSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCLiiaQgMCIsICLiiaQgMCIsICLiiaQgMCIsICIwLjUiLCIxIiwgIj4gMSIpKQpgYGAKCmBgYHtyfQpsZXZlbHMoZmFjdG9yKHZpcnVzZXMka2VlcF9zY29yZV92aXN1YWxpemUpKQpgYGAKCgpgYGB7cn0KcGFsIDwtIGdndGhlbWVzOjp0YWJsZWF1X2NvbG9yX3BhbChwYWxldHRlPSJUYWJsZWF1IDIwIiwgdHlwZT0icmVndWxhciIpCnAxIDwtIGdncGxvdCh2aXJ1c2VzLCBhZXMoeD1hcy5mYWN0b3IoSW5kZXgpLAogICAgICAgICAgICAgICAgICAgZmlsbD1rZWVwX3Njb3JlX3Zpc3VhbGl6ZSwgY29sb3I9a2VlcF9zY29yZV92aXN1YWxpemUpKSArCiAgZ2VvbV9iYXIoc3RhdD0iY291bnQiLCBwb3NpdGlvbj0ic3RhY2siKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgY29vcmRfZmxpcCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNikKICApICsKICBzY2FsZV9jb2xvcl9tYW51YWwobmFtZSA9ICdWaXJhbCBTY29yZScsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKGMocGFsKDQpKSwgMSkpICsKICBzY2FsZV9maWxsX21hbnVhbChuYW1lID0gJ1ZpcmFsIFNjb3JlJywKICAgICAgICAgICAgICAgICAgICAgdmFsdWVzID0gYWxwaGEoYyhwYWwoNCkpLCAwLjUpKSArCiAgeGxhYigiSW5kZXgiKSArCiAgeWxhYigiU2VxdWVuY2UgQ291bnQiKSArCiAgZmFjZXRfd3JhcCh+Y29uZnVzaW9uX21hdHJpeCwgc2NhbGVzID0gImZyZWUiKQpwMQpgYGAKCgojIFJPQyAKYGBge3J9CmxpYnJhcnkocFJPQykKYGBgCgpgYGB7cn0KdmlydXNlcyR0cnVlcG9zaXRpdmUgPC0gcmVwKDAsIG5yb3codmlydXNlcykpCnZpcnVzZXMkdHJ1ZXBvc2l0aXZlW3ZpcnVzZXMkc2VxdHlwZT09InZpcnVzIl0gPC0gMQpgYGAKCgpgYGB7cn0Kcm9jb2JqIDwtIHJvYyh2aXJ1c2VzJHRydWVwb3NpdGl2ZSwgdmlydXNlcyRrZWVwX3Njb3JlKQpyb2NvYmpfYWxsIDwtIHJvYyh2aXJ1c2VzJHRydWVwb3NpdGl2ZSwgdmlydXNlcyRrZWVwX3Njb3JlX2FsbCkKYXVjIDwtIHJvdW5kKGF1Yyh2aXJ1c2VzJHRydWVwb3NpdGl2ZSwgdmlydXNlcyRrZWVwX3Njb3JlKSw0KQphdWNfYWxsIDwtIHJvdW5kKGF1Yyh2aXJ1c2VzJHRydWVwb3NpdGl2ZSwgdmlydXNlcyRrZWVwX3Njb3JlX2FsbCksNCkKI2NyZWF0ZSBST0MgcGxvdApnZ3JvYyhyb2NvYmosIGNvbG91ciA9ICdzdGVlbGJsdWUnLCBzaXplID0gMikgKwogIGdndGl0bGUocGFzdGUwKCdST0MgQ3VydmUgJywgJyhBVUMgPSAnLCBhdWMsICcpJykpICsKICBjb29yZF9lcXVhbCgpCmdncm9jKHJvY29ial9hbGwsIGNvbG91ciA9ICdncmVlbicsIHNpemUgPSAyKSArCiAgZ2d0aXRsZShwYXN0ZTAoJ1JPQyBDdXJ2ZSAnLCAnKEFVQyA9ICcsIGF1Y19hbGwsICcpJykpCmBgYApTZW5zaXRpdml0eTogVGhlIHByb2JhYmlsaXR5IHRoYXQgdGhlIG1vZGVsIHByZWRpY3RzIGEgcG9zaXRpdmUgb3V0Y29tZSBmb3IgYW4gb2JzZXJ2YXRpb24gd2hlbiBpbmRlZWQgdGhlIG91dGNvbWUgaXMgcG9zaXRpdmUuClNwZWNpZmljaXR5OiBUaGUgcHJvYmFiaWxpdHkgdGhhdCB0aGUgbW9kZWwgcHJlZGljdHMgYSBuZWdhdGl2ZSBvdXRjb21lIGZvciBhbiBvYnNlcnZhdGlvbiB3aGVuIGluZGVlZCB0aGUgb3V0Y29tZSBpcyBuZWdhdGl2ZS4KCgoKCiMgQ29tcGFyaW5nIGJlaGF2aW9yIG9mIGFsbCB0ZXN0aW5nIHNldHMgY29tYmluZWQgKGNsdXN0ZXJpbmcgYW5hbHlzZXMpCgpgYGB7cn0KdmlyYWxfc2NvcmVzIDwtIG1hdHJpeChkYXRhPTAsIG5yb3c9bnJvdyh2aXJ1c2VzKSwgbmNvbD1ucm93KGNvbWJvc19saXN0KSkKbnVtX3ZpcnVzZXMgPC0gZGF0YS5mcmFtZSh0b29sY29tYm89cmVwKDAsIG5yb3coY29tYm9zX2xpc3QpKSwKICAgICAgICAgICAgICAgICAgICAgICAgICBudW1fdmlydXNlcz1yZXAoMCwgbnJvdyhjb21ib3NfbGlzdCkpKQoKZm9yIChpIGluIDE6bnJvdyhjb21ib3NfbGlzdCkpIHsKICB2aXJhbF9zY29yZXNbLGldIDwtIGdldHRpbmdfdmlyYWxfc2V0XzEodmlydXNlcywgaW5jbHVkZV92aWJyYW50ID0gY29tYm9zX2xpc3QkVklCUkFOVFtpXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3ZpcnNvcnRlciA9IGNvbWJvc19saXN0JFZTW2ldLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGluY2x1ZGVfdmlyc29ydGVyMiA9IGNvbWJvc19saXN0JFZTMltpXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX3R1bmluZyA9IGNvbWJvc19saXN0JENoZWNrVltpXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbmNsdWRlX2thaWp1ID0gY29tYm9zX2xpc3QkS2FpanVbaV0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaW5jbHVkZV9kZWVwdmlyZmluZGVyID0gY29tYm9zX2xpc3QkRFZGW2ldKQogIAogIG51bV92aXJ1c2VzJG51bV92aXJ1c2VzW2ldIDwtIHRhYmxlKHZpcmFsX3Njb3Jlc1ssaV0+PTEpW1syXV0KICAKICBudW1fdmlydXNlcyR0b29sY29tYm9baV0gPC0gY29tYm9zX2xpc3QkdG9vbGNvbWJvW2ldCiAgCiAgbnVtX3ZpcnVzZXMkdG9vbGNvbWJvMltpXSA8LSBjb21ib3NfbGlzdCR0b29sY29tYm8yW2ldCn0KCm51bV92aXJ1c2VzJG51bXRvb2xzIDwtIHN0cl9jb3VudChudW1fdmlydXNlcyR0b29sY29tYm8sICIxIikKbnVtX3ZpcnVzZXMgPC0gbnVtX3ZpcnVzZXNbb3JkZXIobnVtX3ZpcnVzZXMkbnVtX3ZpcnVzZXMsIGRlY3JlYXNpbmc9RiksXQpudW1fdmlydXNlcyR0b29sY29tYm8gPC0gZmFjdG9yKG51bV92aXJ1c2VzJHRvb2xjb21ibywgbGV2ZWxzID0gdW5pcXVlKG51bV92aXJ1c2VzJHRvb2xjb21ibykpCm51bV92aXJ1c2VzJHRvb2xjb21ibzIgPC0gZmFjdG9yKG51bV92aXJ1c2VzJHRvb2xjb21ibzIsIGxldmVscyA9IHVuaXF1ZShudW1fdmlydXNlcyR0b29sY29tYm8yKSkKbnVtX3ZpcnVzZXMkbnVtdG9vbHMgPC0gYXMuZmFjdG9yKG51bV92aXJ1c2VzJG51bXRvb2xzKQpgYGAKCgpgYGB7cn0KZ2dwbG90KG51bV92aXJ1c2VzLCBhZXMoeD10b29sY29tYm8sIHk9bnVtX3ZpcnVzZXMsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9bnVtdG9vbHMsIGZpbGw9bnVtdG9vbHMpKSArCiAgZ2VvbV9wb2ludCgpICsKICB0aGVtZV9saWdodCgpICsKICB0aGVtZSgKICAgIHBhbmVsLmdyaWQubWFqb3IueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIHBhbmVsLmJvcmRlciA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGF4aXMudGlja3MueSA9IGVsZW1lbnRfYmxhbmsoKSwKICAgIGxlZ2VuZC5wb3NpdGlvbiA9ICJib3R0b20iLAogICAgYXhpcy50ZXh0Lnk9ZWxlbWVudF90ZXh0KHNpemU9MTQpLAogICAgYXhpcy50ZXh0Lng9ZWxlbWVudF90ZXh0KHNpemU9MTQsIGFuZ2xlID0gOTApLAogICAgbGVnZW5kLnRleHQ9ZWxlbWVudF90ZXh0KHNpemU9MTIpLAogICAgYXhpcy50aXRsZT1lbGVtZW50X3RleHQoc2l6ZT0xNiksCiAgKSArCiAgeGxhYigiVG9vbCBDb21iaW5hdGlvbiAoQ1YsIERWRiwgS0osIFZCLCBWUywgVlMyKSIpICsKICB5bGFiKCJOdW0gVmlydXNlcyBQcmVkaWN0ZWQiKQoKZ2dwbG90KG51bV92aXJ1c2VzLCBhZXMoeD10b29sY29tYm8yLCB5PW51bV92aXJ1c2VzLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yPW51bXRvb2xzLCBmaWxsPW51bXRvb2xzKSkgKwogIGdlb21fcG9pbnQoKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0LCBhbmdsZSA9IDkwKSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHhsYWIoIlRvb2wgQ29tYmluYXRpb24gKENWLCBEVkYsIEtKLCBWQiwgVlMsIFZTMikiKSArCiAgeWxhYigiTnVtIFZpcnVzZXMgUHJlZGljdGVkIikKYGBgCgpgYGB7cn0KZ2dwbG90KG51bV92aXJ1c2VzLCBhZXMoeD1udW10b29scywgeT1udW1fdmlydXNlcykpICsKICBnZW9tX2JveHBsb3QoYWVzKGNvbG9yPW51bXRvb2xzKSkgKwogIGdlb21fcG9pbnQoYWVzKGNvbG9yPW51bXRvb2xzLCBmaWxsPW51bXRvb2xzKSkgKwogIHRoZW1lX2xpZ2h0KCkgKwogIHRoZW1lKAogICAgcGFuZWwuZ3JpZC5tYWpvci55ID0gZWxlbWVudF9ibGFuaygpLAogICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpLAogICAgbGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIsCiAgICBheGlzLnRleHQueT1lbGVtZW50X3RleHQoc2l6ZT0xNCksCiAgICBheGlzLnRleHQueD1lbGVtZW50X3RleHQoc2l6ZT0xNCwgYW5nbGUgPSA5MCksCiAgICBsZWdlbmQudGV4dD1lbGVtZW50X3RleHQoc2l6ZT0xMiksCiAgICBheGlzLnRpdGxlPWVsZW1lbnRfdGV4dChzaXplPTE2KSwKICApICsKICB4bGFiKCJOdW1iZXIgb2YgVG9vbHMiKSArCiAgeWxhYigiTnVtIFZpcnVzZXMgUHJlZGljdGVkIikKYGBgCgoKYGBge3J9CnZpcmFsX3Njb3Jlc19ub3plcm9zIDwtIHZpcmFsX3Njb3Jlc1tyb3dTdW1zKHZpcmFsX3Njb3Jlcyk+MCxdCnZpcmFsX3Njb3Jlc19ub3plcm9zIDwtIHZpcmFsX3Njb3Jlc19ub3plcm9zICsgMQp2aXJhbF9zY29yZXNfbm96ZXJvcyA8LSBhcy5kYXRhLmZyYW1lKHZpcmFsX3Njb3Jlc19ub3plcm9zKQoKY29sbmFtZXModmlyYWxfc2NvcmVzX25vemVyb3MpIDwtIG51bV92aXJ1c2VzJHRvb2xjb21ibzIKYGBgCgpgYGB7cn0KbGlicmFyeShwaHlsb3NlcSkKYGBgCgoKYGBge3J9CnRvb2xkYXRhIDwtIG51bV92aXJ1c2VzCgpyb3duYW1lcyh0b29sZGF0YSkgPC0gdG9vbGRhdGEkdG9vbGNvbWJvMgpgYGAKCmBgYHtyfQpwaHlzZXFfcG9vbGVkIDwtIHBoeWxvc2VxKG90dV90YWJsZSh2aXJhbF9zY29yZXNfbm96ZXJvcywgdGF4YV9hcmVfcm93cyA9IFQpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc2FtcGxlX2RhdGEodG9vbGRhdGEpKQpgYGAKCmBgYHtyfQpvcmRpbmF0aW9uIDwtIHBoeWxvc2VxOjpvcmRpbmF0ZShwaHlzZXEgPXBoeXNlcV9wb29sZWQsIG1ldGhvZCA9ICJQQ29BIiwgZGlzdGFuY2UgPSAiYnJheSIpCnBoeWxvc2VxOjpwbG90X29yZGluYXRpb24ocGh5c2VxID0gcGh5c2VxX3Bvb2xlZCwgb3JkaW5hdGlvbiA9IG9yZGluYXRpb24sCiAgICAgICAgICAgICAgICAgICAgICAgICAgc2hhcGU9Im51bXRvb2xzIiwgY29sb3I9Im51bV92aXJ1c2VzIikgKyAKICBnZW9tX3BvaW50KHNpemUgPSAzKSArCiAgdGhlbWVfYncoKSArCiAgZ2VvbV9sYWJlbChsYWJlbD10b29sZGF0YSR0b29sY29tYm8pCgpwaHlsb3NlcTo6cGxvdF9vcmRpbmF0aW9uKHBoeXNlcSA9IHBoeXNlcV9wb29sZWQsIG9yZGluYXRpb24gPSBvcmRpbmF0aW9uLAogICAgICAgICAgICAgICAgICAgICAgICAgIHNoYXBlPSJudW10b29scyIsIGNvbG9yPSJudW1fdmlydXNlcyIpICsgCiAgZ2VvbV9wb2ludChzaXplID0gMykgKwogIHRoZW1lX2J3KCkKYGBgCnRvIGRvOiB0cnkgY29sb3JpbmcgYWJvdmUgYmFzZWQgb24gdGhlIEYxIHNjb3JlcyBvZiB0aGUgdGVzdGluZyBzZXQgb24gZWFjaCBjb21iaW5hdGlvbgoKYGBge3J9CmJyYXlfZGlzdCA8LSBwaHlsb3NlcTo6ZGlzdGFuY2UocGh5c2VxX3Bvb2xlZCwgbWV0aG9kPSJicmF5IikKY2x1c3RlcnMgPC0gaGNsdXN0KGRpc3QoYnJheV9kaXN0KSkKcGxvdChjbHVzdGVycykKCm15Y2x1c3RlcnMgPC0gY3V0cmVlKGNsdXN0ZXJzLCBoPTEuMSkKYGBgCgoKYGBge3J9Cm5hbWVzKG15Y2x1c3RlcnNbbXljbHVzdGVycz09MV0pCm5hbWVzKG15Y2x1c3RlcnNbbXljbHVzdGVycz09Ml0pCm5hbWVzKG15Y2x1c3RlcnNbbXljbHVzdGVycz09M10pCm5hbWVzKG15Y2x1c3RlcnNbbXljbHVzdGVycz09NF0pCm5hbWVzKG15Y2x1c3RlcnNbbXljbHVzdGVycz09NV0pCgpteWNsdXN0ZXJzX2RmIDwtIHRpYmJsZShjb21ibz1uYW1lcyhteWNsdXN0ZXJzKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsdXN0ZXJfaW5kZXg9bXljbHVzdGVycykKCm15Y2x1c3RlcnNfZGYgPC0gc2VwYXJhdGUobXljbHVzdGVyc19kZiwgY29sPWNvbWJvLCBpbnRvPWMoIkNoZWNrViIsICJEVkYiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiS2FpanUiLCAiVklCUkFOVCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJWaXJTb3J0ZXIiLCAiVmlyU29ydGVyMiIpLAogICAgICAgICAgICAgICAgICAgICAgICAgIHNlcD0iICIsIHJlbW92ZSA9IEYpCgoKdG9vbF9jb3VudCA8LSBhcy5kYXRhLmZyYW1lKHJiaW5kKHRhYmxlKG15Y2x1c3RlcnNfZGYkQ2hlY2tWLCBteWNsdXN0ZXJzX2RmJGNsdXN0ZXJfaW5kZXgpWzIsXSwKICAgICAgICAgICAgICAgICAgICAgICAgICB0YWJsZShteWNsdXN0ZXJzX2RmJERWRiwgbXljbHVzdGVyc19kZiRjbHVzdGVyX2luZGV4KVsyLF0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgdGFibGUobXljbHVzdGVyc19kZiRLYWlqdSwgbXljbHVzdGVyc19kZiRjbHVzdGVyX2luZGV4KVsyLF0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgdGFibGUobXljbHVzdGVyc19kZiRWSUJSQU5ULCBteWNsdXN0ZXJzX2RmJGNsdXN0ZXJfaW5kZXgpWzIsXSwKICAgICAgICAgICAgICAgICAgICAgICAgICB0YWJsZShteWNsdXN0ZXJzX2RmJFZpclNvcnRlciwgbXljbHVzdGVyc19kZiRjbHVzdGVyX2luZGV4KVsyLF0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgdGFibGUobXljbHVzdGVyc19kZiRWaXJTb3J0ZXIyLCBteWNsdXN0ZXJzX2RmJGNsdXN0ZXJfaW5kZXgpWzIsXSkKICAgICAgICAgICAgICAgICAgICApCgp0b29sX2NvdW50JG1ldGhvZCA8LSBjKCJDaGVja1YiLCAiRFZGIiwgIkthaWp1IiwgIlZJQlJBTlQiLCAiVmlyU29ydGVyIiwgIlZpclNvcnRlcjIiKQoKdG9vbF9jb3VudCA8LSBtZWx0KHRvb2xfY291bnQpCgpjb2xuYW1lcyh0b29sX2NvdW50KSA8LSBjKCJ0b29sIiwgImNsdXN0ZXJfaW5kZXgiLCAidG9vbF9jb3VudCIpCmBgYAoKYGBge3J9CnBhbCA8LSBnZ3RoZW1lczo6dGFibGVhdV9jb2xvcl9wYWwocGFsZXR0ZT0iVGFibGVhdSAxMCIsIHR5cGU9InJlZ3VsYXIiKQoKZ2dwbG90KHRvb2xfY291bnQsIGFlcyh4PWNsdXN0ZXJfaW5kZXgsIHk9dG9vbF9jb3VudCwKICAgICAgICAgICAgICAgICAgIGZpbGw9Y2x1c3Rlcl9pbmRleCwKICAgICAgICAgICAgICAgICAgIGNvbG9yPWNsdXN0ZXJfaW5kZXgpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgdGhlbWVfbGlnaHQoKSArCiAgdGhlbWUoCiAgICBwYW5lbC5ncmlkLm1ham9yLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBwYW5lbC5ib3JkZXIgPSBlbGVtZW50X2JsYW5rKCksCiAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICBsZWdlbmQucG9zaXRpb24gPSAiYm90dG9tIiwKICAgIGF4aXMudGV4dC55PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGF4aXMudGV4dC54PWVsZW1lbnRfdGV4dChzaXplPTE0KSwKICAgIGxlZ2VuZC50ZXh0PWVsZW1lbnRfdGV4dChzaXplPTEyKSwKICAgIGF4aXMudGl0bGU9ZWxlbWVudF90ZXh0KHNpemU9MTYpLAogICkgKwogIHNjYWxlX2ZpbGxfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNikpLCAwLjUpKSArCiAgc2NhbGVfY29sb3JfbWFudWFsKG5hbWU9IiIsCiAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGFscGhhKHJldihwYWwoNikpLCAxKSkgKwogIHhsYWIoIkNsdXN0ZXIiKSArCiAgeWxhYigiTnVtYmVyIG9mIFRpbWVzIGluIENsdXN0ZXIiKSArIAogIGZhY2V0X3dyYXAofnRvb2wsIHNjYWxlcyA9ICJmcmVlIikKYGBgCgoK